CrowdStrike global outage to cost US Fortune 500 companies $5.4B

$5.4B seems way too low given the number of flights Delta cancelled and will be on the hook to refund.

https://www.washingtonpost.com/transportation/2024/07/23/del...

How does a company this big not have automated tests for their config files, and not have gradual/staggered rollouts for their deployments?

Is there some good reason for this approach (need to get config updates into the wild as quickly as possible to combat zero-days or zero-hours?) or was this just a massive oversight?

Side rant... their postmortem took forever to get to the point, first explaining all their jargon and product names. Makes me really appreciate the Cloudflare ones.

Can't wait for the class action lawsuit. The total impact is likely greater than $5.4B. A significant number of people must have died due to the impact this had on hospitals and emergency services.

Can someone with a background in contract negotiation, vendor onboarding, and business continuity risk management share their expertise? We'd love to hear about typical vendor contract provisions that protect customers in situations like this.

If damages can be demonstrated, what are the chances of airlines successfully claiming compensation? Or, in practice, do such cases usually result in significant discounts during the next contract renewal rather than actual damages paid out?

Don't worry...they're handing out $10 vouchers to make up for it

https://techcrunch.com/2024/07/24/crowdstrike-offers-a-10-ap...

Well, 'security' check boxes have consequences.

Hopefully this will make BigCos think twice about forcing their employees to fill their computers with "security" malware that slows productivity to a crawl.

I think this is going to be a huge boon for dell. We had so many older computers that got completely hosed. Lots of Latitude 5400s died completely. All will need replacements.

Does anyone else feel a little sympathy for CrowdStrike? They pushed out something they should not have. OK. That is bad. But a couple days on and the bulk of the difficulties seem to be from how windows handled the situation: The BSODs, the boot loops, the inability to recover from a basic fault. I feel that if this did happen in a linux environment (it could) that it would be easier to isolate and boot systems into some sort of temporary mode. Linux would communicate and offer options. The windows-specific trend of just abandoning all hope, giving up and throwing the BSOD at the user ... CrowdStrike didn't create that.

But they apologized with a $10 gift card.

Soo. $5.4B - $10

Imagine Apple / Google pushing an update that bricked 2b+ mobile devices.

The preliminary post incident review is here:

https://www.crowdstrike.com/falcon-content-update-remediatio...

It boils down to the "Content Validator" had a bug and gave a false positive.

It's kind of crazy that the 'rapid response content' update was then free to go out direct to production machines with zero actual live testing.

That's either due to c-suite excel cost-cutting/maximize profit or silicon valley yolo.

Is Chaos Engineering an appropriate preventative measure for this sort of thing?

That's a lot of $10 (non functional) Uber eats cards

So, here's 10 bucks, we good?

Bankruptcy coming ....

"We can't hold back releases from going into prod.. we have to deliver"

"We don't have enough time to write tests"

"Developers should be able to test their own code"

Free just got cheaper.

Yeah, I know using free software isn’t a panacea. Still it would be a step in the right direction, plus I could not refrain from the cheap shot at M$ Windows.

It's my understanding that CrowdStrike customers buy that thing to check a box in some security audit, not because it provides any other benefit.

Let's blame bullshit compliance?

Today in "random number pulled out of someone's ass"

Looks good on a resume:

- Wrote code responsible for $5.4 billion

I’m pretty certain CS has contracts that limit their liabilities in events like this.

Probably a refund is all they’ll be on the hook for.

Sadly, damage done like this is just chalked up to an accident, and swept under the rug.