Microsoft has serious questions to answer after the biggest IT outage in history

Apparently Crowdstrike also brought down Linux hosts in the same way in April but it didn’t get widely reported: https://news.ycombinator.com/item?id=41005936

Not sure what questions Microsoft have to answer. A third-party vendor shipped defective software.

I guess the only question they could answer is why they don't provide a framework like Apple do with Endpoint Security for third-party vendors to use.

“The Entire Culture Around So-Called ‘Software Engineering’ And Our Collective Failure to Build Strong Legal Institutions Around That Culture has serious questions to answer after the biggest IT outage in history”

there fixed it for you

I predict that the people who were actually responsible (the "deadline above all else" crowd) will not be the ones who are actually blamed.

I don’t see how this is Microsoft’s fault or issue.

MS can’t prevent a software vendor from breaking the machine.

I see discussion about who's at fault: Microsoft or Crowdstrike.

But one thing I don't get about this: what was the role of the enterprise admins?

Most administrators at large companies are cautious about rolling out new software versions to their employees. They (normally?) test before broad deployment.

Seems like one of three things would have had to have happened for this to be missed:

1. Admins ignored testing this update prior to enterprise rollout.

2. Crowdstrike forced the update on unwilling users.

3. Crowdstrike does not provide a framework for such pre-rollout testing, and enterprises chose to use it anyway.

Can anyone offer insight?

[Disclosure: I'm a Microsoft employee, but not an enterprise admin]

I'm not sure what to make of this but I'm noting something odd this morning: coverage of this event out of the UK near-unanimously is laying this outage on Microsoft. BBC ran a story this morning that didn't mention Crowdstrike until the 4th paragraph, and headline after headline is repeating the message that Microsoft caused a global outage.

Reporting from the US and elsewhere seems to be a bit more on point. Is it just because the Brits went to press earlier in the day before the problem was understood?

I can’t explain enough how much I dislike Microsoft the corporation, but this wasn’t their fault - a 3rd party kernel driver crashed the system.

Well... The only question should be architectural:

- why automatic, silent upgrades

- why no boot environment/generations at boot to reboot into a previous snapshot of the system (since nfts do have snapshots indeed), meaning why no integration between the storage and the system management

- why massive rollout instead of partitioned testing rollout slowly propagating

For the rest is a third party tool, not mandated by the vendor so... It's a user choice.

This is a pretty empty article that just seems like spin on the current issues going on.

The problematic driver was dowloaded from Microsoft managed infrastructure, even though it was a third-party module. Microsoft needs to do a better job at running integration tests between windows kernel and driver updates for sure. They can’t publish security updates without running integration tests, this is basics of software engineering. Windows is their product not Crowdstrike. 100% Microsoft failure here.

I have been 100% Microsoft-free in all computers and networks I control for decades. I know I should miss Excel, etc. but strangely I feel like I have sacrificed nothing. There is more I can do without Microsoft than with it.

The answer is Cloud based Windows OS which MS is working towards for many years.

I'd be curious whether people would want the cost effective fix for this, which is basically to eliminate vendors for anything important.

Near complete vertical integration of security, like with Apple.

What an infuriatingly poor article, Tom Clarke should be ashamed.

> A software update from cybersecurity company CrowdStrike has now taken a large number of those machines offline.

So Tom opens the article with the admission that it is CrowdStrike, not Microsoft.

> Thankfully, the update that caused the Microsoft meltdown did not affect these other software families - if it had, the impacts could have been catastrophic.

This is such a strawman (like the rest of the article honestly) I don't know where to begin. Inflammatory language.

A fucking "meltdown"? A meltdown of Microsoft, no less? Putting aside the fact that Microsoft and Windows are not the same thing, it is again nothing "meltdown" like that Microsoft did or could do.

> There are serious questions of course for CrowdStrike. As a leading provider of security software for large companies like Microsoft.

Tom, was you paid by CrowdStrike or what? What do you mean "of course"? It is literally the only party that should be answering questions here. I suspect that even if Tom were to "question" Microsoft their answer about kernels, drivers, privileges, and how shipping seemingly untested code into the core of an operating system is a bad idea wouldn't even be comprehendible for him.

> The situation may also lead to calls from Microsoft users about what more the company could do to ensure products made for their software aren't going to cause major outages like this one.

This is getting absurd now, and I just can't give more energy to this. OK, it could now insist only memory safe languages such as Rust are allowed for drivers. Or outright permanently blacklisting drivers from certain vendors. The bitching and moaning from manufacturers would then, of course, have people like Tom writing articles like "Microsoft is making manufacturers lives harder, think of the poor IT professionals!".

> Any engineer will tell you over-reliance on one system leaves you open to a "single point of failure". Critical digital infrastructure has to have redundancy - back up systems - built in to ensure it is resilient.

Please Tom, tell us more on your thoughts about memory safe languages, failure recovery modes, the unikernel vs microkernel debate, and how it's just a simple matter of overnight making operating systems "not a single point of failure".

This entire article is some kind of exercise in trying to get everything wrong while meeting a minimum word count, and I bet with some ChatGPT thrown in there too.

I flagged this post because I think it's far below even the minimum quality level for HN. It's outright clickbait drivel.