CrowdStrike broke Debian and Rocky Linux months ago

What gets me is that much of the OSS/Linux ecosystem consists of thousandas of lashed together piles of code written by independent and only very loosely coordinated groups, much of it code and lashed together by amateurs for free, and it is still more robust than software created by multi-billion dollar corporations.

Perhaps one reason is that OSS system programmers are washing their dirty linen in public; not a matter of "many eyes make bugs shallow", but that "any eyes make bad code embarassing".

Just for example, I'm planning to make one of my commercial projects open source, and I am going to have to do a lot of fixing up before I'm willing to show the source code in public. It's not terrible code, and it works perfectly well, but it's not the sort of code I'd be willing to show to the world in general. Better documentation, TODO and FIXME fixing, checking comments still reflect the code, etc. etc.

But for all my sense of shame for this (perfectly good and working) software, I've seen the insides of several closed-source commercial code bases and seen far, far worse. I would imagine most "enterprise" software is written to a similar standard.

Relevant comment from yesterday's Crowdstrike mega-thread:

"Crowdstrike did this to our production linux fleet back on April 19th, and I've been dying to rant about it." [1]

Continues with a multi-para rant.

[1] https://news.ycombinator.com/item?id=41005936

I used to work in this space, and I always had the nagging question of "is any of this stuff actually useful?"

It seems a hard question to answer, but are there any third party studies of the effectiveness of Crowdstrike et al. or are we all making our lives worse for some security theater?

Product quality is on freefall: from aircraft to software. Lack of QA is the norm nowadays as everyone just care about the extra penny.

There was also this report of CrowdStrike injecting a buggy DLL into Windows applications, which could cause the app to crash through no fault of its own:

https://x.com/molecularmusing/status/1808756095860543916

This is all a consequence of firms being able to contract out of consequential liability.

Perhaps we should render such clauses unenforceable, as we do with contracting out of consequential loss of life.

Or at least limit them.

> The update proved incompatible with the latest stable version of Debian, despite the specific Linux configuration being supposedly supported.

> The analysis revealed that the Debian Linux configuration was not included in their test matrix.

This is suspiciously close to actual fraud. They declare they support configuration X, but they actually do not do any testing on configuration X. That's like telling me my car will have seatbelts, but in no place in manufacturing it is ensured the seatbelts are actually installed and work. I think a car maker that does something like that would be prosecuted. Why Crowdstrike isn't? I mean, one thing if they don't support some version of Linux - ok, too many of them, I can get it. But if you advertise support for it without even bothering to test on it - that's at best willful negligence and possibly outright fraud.

“No one noticed” which is a cute way to say that Crowdstrike suppressed the media noticing. The day of the bug, the HN post had comments about how people tried reporting the issue months ago.

Even the article is written as people noticing. So who didn’t notice? Or were the issues not popular enough to be not ignored?

Is anyone here using Crowdstrike, what does it do? I see it referred to as an 'anti-virus'? I have it installed on my work laptops and I see it as a keylogger and activity monitor. "I got nothing to hide", but still bothers me when some corporate super users spy on me.

Anecdote from SWIM: Employer corp supposedly has CS deployed on all endpoints. Been getting away with just running it in a VM with restricted resources and not hearing anything about it. Did notice the VM failing around that time.

Also heard from an ex-coworker in another large corp where IT just gave up on enforcing compliance for Linux endpoints. I wouldn't be surprised if some IT admins effectively adopt a "don't ask don't tell" policy here: If you can figure out the alternative stack by yourself without causing noise or lying about it, you're on your own. It'd certainly make sense if the motivation for enforcement is largely checkbox- compliance.

I wonder just how widespread this kind of admittedly malicious compliance is and how much it contributed to the April incident not being bigger news...

What is cloud strikes unique selling point? Genuine question, because I'd never heard of them before this.

Is it possible that these events had less impact because the damage was less / more easily fixed due to the nature of the OS?

Or perhaps because the admins of Linux systems are typically more knowledable about how to run their platforms, and not just install them?

Or is it due to sheer numbers of enterprise software running on Windows?

Not just Debian and Rocky, but RHEL too. https://access.redhat.com/solutions/7068083

I ran into this doing CentOS7 to Alma9 upgrades. The bug was in RHEL, Alma and Rocky and any other distro derived from RHEL. I had a VM go into an endless reboot cycle and the only way to get back in was to boot to an emergency rescue console and disable falcon-sensor.

The problem was something to do with eBPF, and one of the workarounds was to tell falcon sensor to use kernel mode and not auto or user (bpf) mode.

We don't allow automatic updates on hosts, however, so thankfully this was contained, but it certainly begs the question of just what testing Crowdstrike are doing.

Huh, this story sounds familiar. I read a HN comment the other day telling this same story. They didn't just turn a random HN comment into a news article, did they?

Yup. They did. At least they cited it I suppose.

Need a “crowdstrike.sucks” or maybe a general “it.sucks/{company}” to gather all of these company misgivings.

Avoid this company at all costs. Move to competitors which offer the same “audit passing” requirements

> CrowdStrike should prioritize rigorous testing across all supported

They should not.

Testing costs money and they aren't selling their product to a company that needs or wants it on a competitive market. Their business model is based on shoving the product down the throat of enterprises due to compliance and therefore they have zero incentive to invest any money into quality.

I think someone noticed. And was thinking: People wont be happy to fix this and I am not allowed to fix either. Well, it might be just like the 3000 other rare issues that would one day break the world's IT. Who cares...

In the end, because of regulatory pressure, the only pressure that matters in a commercial environment, there will be three supported OS's: a windows one, a mac one, and probably one, and only one, Linux distribution, or flavor thereof. Everything else will be toast in a commercial environment. For Linux there might be this AWS one, and that Google one, but they'll be close. And, in order to satisfy regulatory requirements, they'll be very, very close. Commercial organizations have bosses and, more ominously, regulators. We, and they, need a checkbox checked. So let's not fool ourselves with thoughts of freedom and liberty. There's a real world out there.

CrowdStrike screwed up, but there's more chance that a 1000 linux's go to one than 1 CrowdStrike goes to zero.

> ( ... ) experienced significant disruptions as a result of CrowdStrike updates, raising serious concerns about the company's software update and testing procedures

To me the issue isn't CrowdStrike's testing procedures. To me the issue is why does Debian depend on CrowdStrike? Does anyone understand this?