Programming

Hey, why not just ask Dave Plummer, former Windows developer...

https://youtube.com/watch?v=wAzEJxOo1ts

When anywhere from 8.5 million to over a billion systems went down, numbers I've read so far vary significantly, still that's way too much failure for a simple borked update to a kernel level driver, not even made by Microsoft.

That means spending time and money on developing such a system, which means increasing costs in the short term.. which is kryptonite for current-day CEOs

I strongly believe in no-blame mindsets, but "blame" is not the same as "consequences" and lack of consequences is definitely the biggest driver of corporate apathy. Every incident should trigger a review of systemic and process failures, but in my experience corporate leadership either sucks at this, does not care, or will bury suggestions that involve spending man-hours on a complex solution if the problem lies in that "low likelihood, big impact" corner.
Because likely when the problem happens (again) they'll be able to sweep it under the rug (again) or will have moved on to greener pastures.

What the author of the article suggests is actually a potential fix; if developers (in a broad sense of the word and including POs and such) were accountable (both responsible and empowered) then they would have the power to say No to shortsighted management decisions (and/or deflect the blame in a way that would actually stick to whoever went against an engineer's recommendation).

Edited the title to have a by in front to make that a bit more clear

Is that the case at CrowdStrike?

Weren't the issues at airports because of the ticketing and scheduling systems going down, not anything with aircraft?

That's just covering up, like a disclaimer that your software is intended to only be used on 29ᵗʰ of February. You don't expect anyone to follow that rule, but you expect the court to rule that the user is at fault.

Luckily, it doesn't always work that way, but we will see how it turns out this time

Yeah exactly. You'd think they'd have a test suite before pushing an update, or do a staggered rollout where they only push it to a sample amount of machines first. Just blaming one guy because you had an inadequate UAT process is ridiculous.

OMG the article conflates kennel API calls and kennel drivers such as what crowdstrike actually does. I refuse to read it until the end.

Check Crowdstrike's blurb about the 1-10-60 rule.

You can bet that they have a KPI that says they can deliver a patch in under 15m; that can preclude testing.

Although that would have caught it, what happened here is that 40k of nuls got signed and delivered as config. Which means that unparseable config on the path from CnC to ring0 could cause a crash and was never covered by a test.

It's a hell of a miss, even if you're prepared to accept the argument about testing on the critical path.

(There is an argument that in some cases you want security aystems to fail closed; however that's an extreme case - PoS systems don't fall into that - and you want to opt into that explicitly, not due to a test omission.)

The update that crashed things was an anti-malware definitions update, Crowdstrike offers no way to delay or stage them (they are downloaded automatically as soon as they are available), and there's good reason for not wanting to delay definition updates as it leaves you vulnerable to known malware longer.

Four days for an update to malware definitions is how computers get infected with malware. But you're right that they should at least do some sort of simple test. "Does the machine boot, and are its files not getting overzealously deleted?"

Real question is: what’s wrong with the test suites?

This is what I'm asking myself too. If they tested it, and they should have, then this massive error would not happen: a) controlled test suites and machines in their labors, b) at least one test machine connected through internet and acting like a customer, tested by real human, c) update in waves throughout the day. They can't tell me that they did all of these 3 steps. -meme

Therefore this it is the company’s responsibility as a whole.

The governance of the company as a whole is the CEO's responsibility. Thus a company-wide failure is 100% the CEO's fault.

If the CEO does not resign over this, the governance of the company will not change significantly, and it will happen again.