this post was submitted on 22 Jul 2024
586 points (97.7% liked)

Technology

58303 readers
13 users here now

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related content.
  3. Be excellent to each another!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, to ask if your bot can be added please contact us.
  9. Check for duplicates before posting, duplicates may be removed

Approved Bots


founded 1 year ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
[–] [email protected] 11 points 3 months ago (5 children)

In April, a CrowdStrike update caused all Debian Linux servers in a civic tech lab to crash simultaneously and refuse to boot.

And then, you boot their servers from a Linux Live USB, run TimeShift to restore the last system snapshot, refuse the latest patch from Cloudstrike and they all lived happily ever after.

[–] [email protected] 23 points 3 months ago

None of these things are used in actual server operations.

[–] [email protected] 22 points 3 months ago

And it's not much more difficult to fix on Windows, except for the scale of the problem.

[–] [email protected] 12 points 3 months ago (1 children)

Good luck doing that remotely. Which is the sole problem with this most recent CrowdStrike bug.

[–] [email protected] 1 points 3 months ago

Anybody who doesn't already have ipmi serial console access set up needs to put that on their list of acceptance criteria for remediation of this incident.

[–] [email protected] 4 points 3 months ago

And on Windows you booted in safe mode and removed one file. What's the point of your post?

[–] [email protected] 2 points 3 months ago

boot their servers from a Linux live usb

If I ran a computer lab that wasn't already net booted, I'd use this as the motivating factor to put that in place. Net booting to a repair image, or just reinstalling the whole OS either from scratch or a known good disk image, is where anybody who manages a fleet of computers should be.

There was a point in time where I had a pxe boot server vm set up on my laptop that I used to reload servers in our little row of racks at 365 main, because it let me quickly swap out the boot iso, and was faster than usb sticks were at the time.