The CrowdStrike Failure Was a Warning

The correct lesson is to stop introducing more vulns into your systems by running "security" products. Crowdstrike was just an outage but could have just has easily been Solarwinds 2.0.

Crowdstrike is probably less bad than the alternatives that I have run into that are largely developed by very low cost engineers cough TrendMicro cough but even so, they aren't NT kernel engineers nor do they have the NT kernel release process.

Companies need to find ways to live without this crap or this will keep happening and it will be a lot worse one day. Self-compromising your own systems with RATs/MDMs/EDR/XDR/whatever other acronym soup needed to please the satanic CISSPs are just terrible ideas in general.

You are only as good as your weakest link:

> A Microsoft spokesman said it cannot legally wall off its operating system in the same way Apple does because of an understanding it reached with the European Commission following a complaint. In 2009, Microsoft agreed it would give makers of security software the same level of access to Windows that Microsoft gets.

https://www.wsj.com/tech/cybersecurity/microsoft-tech-outage...

I found the article unbearable and just a convoluted way to say: this incident would have had a lot less impact if CrowdStrike had less customers or more competitors. A real page filler without any insight or solutions, just look at this paragraph, completely void of anything useful.

> This time, the digital cataclysm was caused by well-intentioned people who made a mistake. That meant the fix came relatively quickly; CrowdStrike knew what had gone wrong. But we may not be so lucky next time. If a malicious actor had attacked CrowdStrike or a similarly essential bit of digital infrastructure, the disaster could have been much worse.

Gee, the damage from an honest mistake (what does the author even base that on) is most likely easier to fix than the damage done by a malicious actor with bad intent. I feel so enlightened!

The warning in this case is hire security people who actually have a clue and include vendor software in their risk assessment.

Literally every time I see stuff like this go down, the security software had exactly zero engineering research put into it whereas everything else did.

If people did this, CrowdStrike would either not exist or look completely different.

It was a seizure, not warning. More stupid bandaids will be slapped in a hurry without considerations from people who understand how this tumor works.

Having monopolies and oligopolies is like having a small gene pool.

The culture at MS$ is to servitize enough of their products and then force customers to use them. That way, the products won't be so exposed to users and Microsoft will be able to limit their own liability without actually improving the back end product.

Servitization is a clever way to consolidate your perpetual licensed customers over to perpetual service contacts, while also further obfuscating and locking down the underlying operating environment.

This is in the best interest of Microsoft bottom line, at the expense of all private business, government, or anyone who values consumer experience really. It reduces the number of drive-by security incidents, but when WW3 happens and 75% of our economy is hosted in a whopping 12 datacenters across 3 companies I'm sure we'll be screwed. I mean just depth charging Google fiber today would probably take down 25% of the world economy.

"Was a Warning."

No. It was just a failure. The warnings have been trumpeted for decades.

It should have been no surprise that the giant company that was trusted to secure our single source of OS software against "supply chain attacks" ended up committing the largest "supply chain attack" yet seen on Earth.

We are effectively still in the wild west. The gold rush has to end before we can truly civilize the place.

is it true that the owner of crowdstrike is an ex-employee of McAfee and the same company that got sold because they had massive downtime for basically the same reason

The article advocates for even more market fragmentation? Even though that isn't the issue at all?

If these machines were backend systems (which most of the ones that mattered were), you have to ask: why are they running malware detection when they should have minimal-to-no surface area for an attacker?

That's the real question here.

Sorry, but if decades of warnings from qualified security actual experts who were hired specifically for their expertise in such matters went ignored enough for long enough to reach this point, then this incident isn't gonna change much anything. It'll be news for a short while, then forgotten. No lessons will have been learned, and few if any changes will be made. More things like this will happen in the future. Guaranteed...

I'm a grumpy old man on the internet.... let's just get that out of the way

The root cause is NOT capitalism, nor is it users, Microsoft, or even CrowdStrike. You can't legislate, regulate, or "be more careful next time" your way out of this. Hell, blaming the users won't even work.

Here are 3 stories:

---

Imagine yourself as an inspector for the Army. The 17th Fortress has exploded this month, and nobody can figure out why. You've checked all the surviving off-site records, and are reasonably sure that the crates of dynamite that used to make up the foundations and structure of the cart were properly inspected, and even updated on a regular basis.

You more closely inspect the records, looking for any possible soldier or supplier who might have caused this loss. It might possibly be communist infiltration, or one of those pacifists!

You encounter an old civilian, who remembers a time when forts were built out of wood or bricks, and suggests that. But he's not a professional solder, what could he know.

---

Imagine you're a fire inspector. You've been to your 4th case this month of complete electrical network outage. This time, the cause seems to be that Lisa Douglas at Green Acres had Eb Dawson climb the pole, and he plugged in one too many appliances to the electricial.

If only there were a way to make sure that an overload anywhere couldn't take down the grid, and ruin so many people's days. You desperately want a day without house fires, and so many linemen being called out to test and repair circuits before connecting them back to the grid.

It will take some time before the boilers and generators get back on line from their cold re-start. In the mean while, business in town has ground to a halt.

The paperwork and processes to track and certify each appliance doesn't seem efficient.

There's this grumpy old guy who talks about fuses and circuit breakers, but he's just a crank.

---

The United States found itself embedded in yet another foreign entanglement in VietNam. There was a severe problem planning air strikes, because there were multiple sources required to plan them, and no single computer could be trusted with both of them. The strikes themselves were classified, but the locations of the enemy radar installations couldn't be trusted to the computers, because they were occasionally accessed by enemy sources. Thus the methods and means of locating the enemy radar equipment could become known, and thus rendered ineffective.

A study was done[1], and the problems were solved. There were systems based on the results of these studies[5], and they worked well.[2] Unfortunately, people thought that it was un-necessary to incorporate these measures, and they defaulted to the broken ambient authority model we're stuck with today. Here's some more reading, if you're interested.[3]

---

If you're bored... I've even got a conspiracy theory that explains how I think we actually got here, it it wasn't simply historical forces (which I think it was, 95% certainty).[4] If true, those forces would still be here today, actively suppressing any such stories.

[1] https://csrc.nist.rip/publications/history/ande72.pdf

[2] https://srl.cs.jhu.edu/pubs/SRL2003-02.pdf

[3] https://github.com/dckc/awesome-ocap

[4] https://news.ycombinator.com/item?id=40107150

[5] https://web.archive.org/web/20120919111301/http://www.albany...

What baffles me is just how many IT personnel in so many organizations around the world apparently just blindly hit the "Deploy this zero-day update to all production systems without any testing" button instead of the "Test this update on our test systems first" button.

Or maybe even just looking up the update online to see whether any problems had been reported before deploying it wholesale across their organizations.

Are these the same IT people whose systems all went offline in the left-pad incident because they 'accidentally' set their production servers to be dependent on a third-party repository?

I've worked at some low-budget places that didn't have much in the way of a vetting process, but even there auto-deploying unknown updates to third-party dependencies into production was always a capital N No.