55
systemd Talks Up Automatic Boot Assessment In Light Of The Crowdstrike-Microsoft Outage
(www.phoronix.com)
this post was submitted on 22 Jul 2024
55 points (95.1% liked)
Linux
5165 readers
207 users here now
A community for everything relating to the linux operating system
Also check out [email protected]
Original icon base courtesy of [email protected] and The GIMP
founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
It was a config file. The CrowdStrike code would already have been in the kernel for quite some time. Would you not need a previous version of the system without those kmods (or whatever they're using)? That is unlikely.
Yeah, you'd need to snapshot their data directory and roll that back. The previous kernel module may well have had the bug already, just not a malformed config file to trip it.
Also, if the driver booted ok, but then panicked soon after, would that count as a bad boot? The description seems to indicate the boot counters get reset as soon as a boot succeeds.
As someone else pointed out, even on an immutable system where you can swap out the system layer, the update would have likely been somewhere in the mutable
/vardirectory in the userspace, since it was some kind of definition update.I believe SteamOS uses ABA to ensure continuous operation in the case of a bad update, but an image rollback would only work if you could include the offending file/directory for anything that's not in the system layer.
I think having an A partition and a B partition (I'm assuming that's how SteamOS works) wouldn't help in this case. If the A partition downloaded the definition file, crashed and failed to reboot; the bootloader could failover to the B partition - which would then download the definition file, crash and fail to reboot. It would have to keep rolling back to a last known good snapshot until the update got withdrawn.
You could have an ephemeral set up that wipes
/varand/etcand recreates them every boot. I don't think these EDR tools would like that very much though.You could potentially block your network by disabling your router or something, so it couldn't download the bad update, but you'd have to know that was a step to prevent it (which most people didn't until it was too late).
Ostree-based systems are handy for replacing the system layer, but configs live (mostly) in userspace, and they persist.
Well at that point, just don't install any kernel mode EDR software at all.
NixOS can be set up for impermanence where all config is recreated every boot and nothing persists besides the nix store. There's helpers for ephemeral home also, so you can have something like TailsOS. I'm sure you could do that with other distros but you'd need absolute discipline to have everything the machine needs provisioned at boot.
Maybe it could run a recovery program on repeated boot failure