July 20th, 2024

CrowdStrike debacle provides road map of American vulnerabilities to adversaries

A national digital meltdown caused by a software bug, not a cyberattack, exposed network fragility. CrowdStrike's flawed update highlighted cybersecurity complexity. Ongoing efforts emphasize the persistent need for digital defense.

Read original articleLink Icon
CrowdStrike debacle provides road map of American vulnerabilities to adversaries

A recent national digital meltdown affecting airports, hospitals, and TV stations was caused by a bug in a software update, not by a nation-state attack as feared. The incident highlighted the fragility of interconnected networks and the challenges in achieving cyberresilience. The flawed update came from CrowdStrike, a tool meant to combat cyberattacks, further emphasizing the complexity of cybersecurity. Efforts to address such vulnerabilities have been ongoing, with government partnerships established to share insights and agencies issuing alerts on cyber threats. The incident serves as a reminder of the persistent challenges in securing digital infrastructure, with the risk of unintended consequences and potential political motives behind future cyber disruptions. Despite advancements in technology like artificial intelligence aiding in identifying vulnerabilities, the process of enhancing cybersecurity remains gradual. The incident underscores the ongoing threat of cyber disruptions and the need for continuous efforts to strengthen digital defenses.

Related

Microsoft outage: Chaos as internet down and flights grounded around the world

Microsoft outage: Chaos as internet down and flights grounded around the world

A global IT outage, possibly linked to Crowdstrike antivirus software, caused chaos worldwide. Windows crashes affected sectors like healthcare and transportation. Crowdstrike's shares dropped. Various services faced disruptions, prompting calls for system modernization.

Global IT Collapse Puts Cyber Firm CrowdStrike in Spotlight

Global IT Collapse Puts Cyber Firm CrowdStrike in Spotlight

A faulty patch from CrowdStrike Holdings Inc. caused a global IT collapse, impacting various sectors. CrowdStrike's shares dropped by 15%, losing $8 billion. The incident emphasized the importance of endpoint protection software.

Microsoft has serious questions to answer after the biggest IT outage in history

Microsoft has serious questions to answer after the biggest IT outage in history

The largest IT outage in history stemmed from a faulty software update by CrowdStrike, impacting 70% of Windows computers globally. Mac and Linux systems remained unaffected. Concerns arise over responsibility and prevention measures.

It's not just CrowdStrike – the cyber sector is vulnerable

It's not just CrowdStrike – the cyber sector is vulnerable

A faulty update from CrowdStrike's Falcon Sensor caused a global outage, impacting various industries. Stock market reacted negatively. Incident raises concerns about cybersecurity reliance, industry concentration, and the need for resilient tech infrastructure.

2024 CrowdStrike incident: The largest IT outage in history

2024 CrowdStrike incident: The largest IT outage in history

A faulty update by CrowdStrike led to a global computer outage affecting airlines, banks, hospitals, and government services. Over 3,200 flights were canceled, emphasizing the need for strong cybersecurity.

Link Icon 36 comments
By @neonate - 3 months
By @ScottBurson - 3 months
Really interesting to me that none of the commentators I've seen in the press have even hinted that maybe an OS that requires frequent security patches shouldn't be used for infrastructure in the first place. For just one example, I've seen photos of BSODs on airport monitors that show flight lists -- why aren't those built on Linux or even OpenBSD?

Security is not a feature that can be layered on. It has to be built in. We now have an entire industry dedicated to trying to layer security onto Windows -- but it still doesn't work.

By @hilbert42 - 3 months
"What Happened to Digital Resilience?"

Was there ever such a time? If so then tell me when it was.

"The latest chaos wasn’t caused by an adversary, but it provided a road map of American vulnerabilities at a critical moment."

I've no doubt that road maps of American vulnerabilities are currently being planned, roadmaped and stockpiled for future use by those who aren't on the best terms with the US.

In one way I'm amazed at how laxadasical the US and others are towards these threats and that they have not done more to harden the vulnerabilities. On the other hand, it's obvious: cost is one factor but I reckon another bigger one is 'convenience'. Hardening systems against vulnerabilities means making them less convenient/easy to use and people instantly balk against that.

Remember, this happened big-time when Microsoft introduced Windows especially Windows 95. To capture the market Microsoft made everything as easy as possible for nontechnical users—just click on something and it'd happen, things would happen with ease. And all this happened without due consideration to security.

When viruses, vulnerabilities, breaches got out of hand restrictions were introduced which meant users had less freedom to do what they'd gotten used to doing. What Microsoft did was to get the world used to slack operating procedures and efforts reign this in has met with user resistance ever since.

We're now stuck with a major problem that was easily foreseeable even before Microsoft launched Windows 95. Fixing it will be extremely difficult.

By @notepad0x90 - 3 months
In a twisted way, Crowdstrike just gave western civilization a disaster recovery and resilience forced test. an actual attack won't be rolled back within an hour.

In case you don't know, Crowdstrike is hardly the only company with large scale access to this many companies,governments and resources. It takes one rogue employee to deploy a disk wiper that destroys every computer (including linux and macos) and affected systems won't recover at all. it would be months before critical systems are back online, the global economy would come to a halt worse than how it did with COVID in such a scenario.

It isn't "why didn't Crowdstrike do better" (although they should have), it is more, why isn't technology in critical systems more resilient to one vendor screwing up or getting hacked?

For example, let's say it wasn't just a boot loop but a disk wiper erased every boot disk, is there any reason pxe booting a recovery image or a backup image configured already on servers, atms, kiosks, point of sale systems,etc...? even if UEFI and bios were erased, it is technically not impossible to have an auto-recovery mechanism implemented right?

If you have never been in an incident response (IT and security incidents) root cause analysis, I don't blame you for not thinking deeper about the root cause, but that is the type of root cause analysis that has been missing despite over a decade of rampant ransomware, disk wipers, and supply chain risks.

Finding someone to blame and be angry at is easy and doesn't solve the root cause. Making hard technical decisions and not wasting this opportunity (never waste a good crisis) to push for resilient technology investments actually solves the root cause behind this and other repeating problems.

By @lambdaone - 3 months
This has been an open secret for decades. Just a handful of major OS and browser vendors, constantly shipping patches to their systems and most software having such vast software supply chains that it's effectively impossible to audit anything, let alone truly certify anything as safe, and "security" software just expands the attack surface.

Everyone in the industry knows this.

Interesting to see the NYT just catching up.

By @sschueller - 3 months
If you are a non-US company you have to be insane to use this CrowdStrike service. The FBI can legally use a secret warrant[1] and force CrowdStrike to inject a DLL into your infrastructure!

[1] https://en.wikipedia.org/wiki/United_States_Foreign_Intellig...

By @encoderer - 3 months
Just told my family yesterday that if we are ever in a real war expect everything to stop working within 8 hours. We will go back to cash and paperwork but it will be painful and slow.
By @AlbertCory - 3 months
"Diversity" (but not in the sense of marginalized people)

If more of the critical machines were running different OS's, the damage would be contained.

When we talk about the dangers of "monoculture" it's usually about plants. The same danger applies to computing infrastructure.

By @simpaticoder - 3 months
Massive computer outage, worldwide affecting enterprises with Windows machines running CrowdStrike, a very popular software that is sold as hacking protection but which is, in reality, used by C-suite execs to spy on employee behavior. It is installed with extraordinary permissions and is difficult to fix or remove by design.

I wonder if this will teach absolutely anyone a lesson about anything.

By @pm90 - 3 months
> It is not hopeless.

> “We are optimistic that A.I. is actually allowing us to make significant — not transformative yet, but significant — progress in being able to identify vulnerabilities, patch holes, improve the quality of coding,” Kent Walker, the president for global affairs at Google, said at the Aspen forum.

I disagree. If the only hope is some vague promise of bs AI, there is no hope indeed.

By @gmuslera - 3 months
There are some point where you should redefine what it mean to be an adversary. To be practically forced into a position that lead to this level of harm, by actors that you don't want to perceive, is something that you may want to analyze.

The purpose of a system is what it actually do, not what it claims to do but fails every time at that. Turning everything to vulnerable as fragile with some big strategic and global plan ahead makes you into a disposable asset, a sacrificial victim in some higher level chess game. And you can agree with that with your decisions.

By @GeoAtreides - 3 months
Here's an interesting exercise: what's the minimum quantity of explosives that would lead to 1% drop in western GDP? would doubling it lead to 2% or 4%? is the relationship linear?

I don't have an answer, but thinking about it makes one understand how incredible fragile our complex logistic chains (and indeed our economy) are. One day all this complexity will collapse upon itself and we'll wonder what happened.

By @cmrdporcupine - 3 months
How long before our evident incompetence as a profession comes back to bite us in the form of more draconian regulation about who and what is allowed to run in kernel space, or other privileged contexts, on critical infrastructure?
By @thedataexchange - 3 months
1. It’s a good time to reread the article that got Dan Geer famous on “monocultures” => https://ccianet.org/wp-content/uploads/2003/09/cyberinsecuri...

2. Also a great time to start prepping for AI Incidents => https://thedataexchange.media/ai-incident-response/

By @oneplane - 3 months
There is no "Digital Resilience" because that is perceived as too expensive, a cost center with hard to quantify value. So it's easier to try and carve out everything that doesn't fit into a spreadsheet, everything that isn't core business, and everything that is not able to present what value it generated.

If general IT had the abilities of sales, marketing, or insurance, there might be a chance that the business would take the responsibility to have the internal knowledge and capabilities to assert control over their systems. But they don't, and as such they won't and instead shove that responsibility over to a third party generalist elsewhere with enough paperwork to have both parties feel their asses are covered.

As long as everything seems to be working, the signals that are still getting through is project failures, be it complete failures or just time and/or money being consumed more than planned and maybe some requirements getting cut. But as soon as enough stuff breaks at the same time, we get news outlets writing articles about resilience and the greater public suddenly no longer agreeing with that is effectively just the result of the status quo because it impacts them directly.

By @mistrial9 - 3 months
externalizing a threat, from a national news source.. Thought experiment -- a healthy society has plural viewpoints, and plural economic strengths. What if a core and entitled group of groups imposed their "security" on a plural society, for their own profit at the expense of the majority? What if their security is monoculture and internally inconsistent, without the ability to admit error ? What if there is a reflex to blame external groups specifically to divert attention from an internal and unbalanced chain of actions, controls and monetary flows?

What is the response of a Free Press to news stories exercising reflexive blame-game from allied core groups with major monetary interests in the outcomes?

By @cdchn - 3 months
If CrowdStrike's system wasn't able to prevent a kernel driver thats all zeros from getting by, you can be sure a malicious payload would have breezed right through.
By @mrjin - 3 months
The fire more deadly than enemy fire is friendly fire. For adversaries, they cannot do any harm unless they get in, even if they get in, the damage is limited to the access of the account they run on. But for AVs, they are invited in, which renders the 1st line of defense useless. Making it worse, they are running with SYSTEM privileges, which is higher than Admin privileges. And we just witnessed what could happen if AVs went rogue.
By @Timber-6539 - 3 months
The only vulnerability here was CrowdStrike's EDR product that runs exclusively in ring 0 and the entire corporate & technical class that lazily relied on this flawed security model and centalized this incompetence.

As much as some people want to believe that Microsoft is blameless here, I hold them partly responsible. They need to create a stable API in their kernel and force third party security vendors to use it.

By @shirro - 3 months
I haven't worked in a Windows environment for a long time so was a little surprised how much of the online commentary suggests people in that environment are comfortable or at least resigned to the necessity of unattended live third party updates on critical infrastructure. I can't see any justification for that on the *nix side of things and hope that culture never transfers over.
By @SkyPuncher - 3 months
For the non-tech folks, this probably felt like one step away from an attack from an adversary.

I have a different take. This was still far from being an adversarial attack. There was no security breach. The failed configuration came from an SDLC that remained secure and fully in control of CrowdStrike. It was a terrible bug, but it was not an attack

By @enceladus06 - 3 months
Why does IT even pay $$$ for crowdstrike? Time to uninstall it and figure something else out. Just use linux or chromeOS.
By @someonehere - 3 months
Can someone from one of the major services comment on why they don’t run the N -1 policy on Falcon? My onboarding sales engineer recommended this to me years ago to avoid this situation. Why do critical infrastructure companies run bleeding edge updates like this?
By @johanneskanybal - 3 months
So yea let’s not use a company like this as best practice. Everything about this reeks of worst practices rising the wave of regulatory capture.
By @m3kw9 - 3 months
the problem is that for a security scanner to scan threats properly, they need to sit on the kernel, there should be a mode where they allow scanners to read but is not able to crash the system. Some sort of sand box for all these kernel access
By @FerretFred - 3 months
Kent Walker's betting the farm on AI spotting future f*ckups? One born every minute!
By @siliconc0w - 3 months
Really, the problem is that all this critical infrastructure runs on Windows. Critical systems should effectively be appliances that run with a very minimal footprint. If you absolutely need to monitor them you can export disk snapshots or something out of band that can't impact operations.
By @bell-cot - 3 months
On the one hand - you can read this as a PSA for the apathetic and/or clueless 99.9%.

On the other hand - it's d*mn hard to imagine that any of America's "A List" or "B List" adversaries didn't have a far-more-detailed road map, years ago.

By @dgoldstein0 - 3 months
Crowdstrike has really redefined malicious compliance
By @naveen99 - 3 months
Or it’s just a front for the nsa and cia.
By @ndesaulniers - 3 months
Yeah, didn't the US just ban Kaspersky, over fears that Kaspersky could cause such an outage (among other fears)?

Turns out our homegrown CrowdStrike was just as bad as our fears over Kaspersky were. Perhaps worse.

By @bitexploder - 3 months
Umm, they (adversaries) already knew? Been in cybersecurity for 18 yrs. We told customers about issues like this all the time
By @linearrust - 3 months
> It was, by all appearances, purely human error — a few bad keystrokes that demonstrated the fragility of a vast set of interconnected networks in which one mistake can cause a cascade of unintended consequences.

Cute. It's always those bad keystrokes. If only these crowdstrike employees worked on their good keystrokes that morning. I blame management.

> Russian hackers working on behalf of Vladimir V. Putin bring down hospital systems across the United States. In others, China’s military hackers trigger chaos, shutting down water systems and electric grids to distract Americans from an invasion of Taiwan. ... Among Washington’s cyberwarriors, the first reaction on Friday morning was relief that this wasn’t a nation-state attack. For two years now, the White House, the Pentagon and the nation’s cyberdefenders have been trying to come to terms with “Volt Typhoon,” a particularly elusive form of malware that China has put into American critical infrastructure.

So we have cyberwarriors and cyberdefenders? And the russians, china, etc have 'hackers'. If ever there was a doubt what the nytimes really is.

> The fear is, in an election year, that the next digital meltdown may have a deeper political purpose.

Oh dear. More bad keystrokes on the way?

Did anyone glean anything of value from the article? There was a lot of words but no substance.

By @adfm - 3 months
This piece was written by someone covering national security and the Biden administration for the NYT. It’s a global issue exposing vulnerabilities across the board. It’s journalism like this that’s the real vuln. Word.
By @pdimitar - 3 months
Wouldn't any memory-safe language help prevent this NULL pointer access? Why are all these crucial pieces still written in C/C++, when it's obvious to anybody keeping even remote track of CVEs that these languages are just not up to the task with today's climate of a 24/7 shadow internet war? (The one that's likely been going on for at least 25 years at this point?)

When will we learn?

You hate Rust -- fine (not fine but OK, I guess people get super triggered over it and it's a reality I can't change but I am still baffled by it because they throw away reason for emotions and these people should really know better). Fine. Just use Golang or any other GC language really (Java or C# as well, if you must).

When will we abandon convenient routine and start adapting to modern realities? ("Modern" being at least 25-year old here but hey, I am willing to give you some leeway and not roast you too much. Let's assume these are "modern" realities, f.ex. just the last 5 years.)