I can't believe a third party company was able to take down so many Windows computers

RiveDroite

I'm sure IT departments worldwide are scrambling today...

https://www.theregister.com/2024/07/19/crowdstrike_falcon_sensor_bsod_incident/

Aaron

Windows has become a public nuisance

DoctorG

Never trust Windows driver updates!

maarten666

This post is deleted!

Pathduck

What's absurd is these companies letting software updates directly into production systems without first testing them. It's basically like doing Russian roulette.

Yes, they are updates to critical security components, but things should always be tested first even if takes a little longer to get updates into production.

DoctorG

We all know why some security experts call such "security" software Snakeoil.

Catweazle

@DoctorG, in any case, the Crowdstrike dev, along with the company, is going to have a very bad day, facing a lot of money for the compensation that MS and the affected companies for sure are going to demand.

alt text

Pathduck

@Catweazle said in I can't believe a third party company was able to take down so many Windows computers:

in any case, the Crowdstrike dev, along with the company, is going to have a very bad day, facing a lot of money for the compensation that MS and the affected companies for sure are going to demand.

There will be lawsuits, for sure. But these companies are not stupid, they have clauses in their contracts to minimize the risk of liability.

"Despite the losses companies have suffered, CrowdStrike was said to be only minimally liable for the damage or lost revenue caused.[58] The terms for CrowdStrike's Falcon software limits liability to 'fees paid',[59] so the maximum compensation an affected company could recover were the fees that the company has paid to CrowdStrike.[60]"
https://en.wikipedia.org/wiki/2024_CrowdStrike_incident#Impact

And I would actually agree, like I said the main fault lies with the companies allowing Crowdstrike to push their software update directly onto production machines without going through testing. These are huge global companies, there's simply no excuse for such a lax attitude to software updates.

But Crowdstrike will lose customers that's for sure, even go bankrupt

And before the Linux users get too smug about this, Linux is in no way immune to such indidents:
https://en.wikipedia.org/wiki/CrowdStrike#Severe_outage_incidents

Only reason this hit so hard is that it mostly hit client systems and those are running Windows all over.

DoctorG

HAHA, thy exclude some use.

8.6 Haftungsausschluss.
MIT AUSNAHME DER AUSDRÜCKLICHEN GEWÄHRLEISTUNGEN IN DIESEM ABSCHNITT 8 LEHNEN CROWDSTRIKE UND SEINE VERBUNDENEN UNTERNEHMEN ALLE ANDEREN GEWÄHRLEISTUNGEN AB, OB AUSDRÜCKLICH, STILLSCHWEIGEND, GESETZLICH ODER ANDERWEITIG. CROWDSTRIKE UND SEINE VERBUNDENEN UNTERNEHMEN UND LIEFERANTEN LEHNEN IM GRÖSSTMÖGLICHEN NACH GELTENDEM RECHT ZULÄSSIGEN UMFANG AUSDRÜCKLICH ALLE STILLSCHWEIGENDEN GEWÄHRLEISTUNGEN DER MARKTGÄNGIGKEIT, DER EIGNUNG FÜR EINEN BESTIMMTEN ZWECK, DER RECHTEINHABERSCHAFT UND DER NICHTVERLETZUNG VON RECHTEN DRITTER IN BEZUG AUF DIE CROWDSTRIKE-ANGEBOTE UND CROWDSTRIKE-TOOLS AB. ES GIBT KEINE GEWÄHRLEISTUNG, DASS DIE ANGEBOTE ODER CROWDSTRIKE-TOOLS FEHLERFREI SIND ODER DASS SIE OHNE UNTERBRECHUNG FUNKTIONIEREN ODER BESTIMMTE ZWECKE ODER BEDÜRFNISSE DES KUNDEN ERFÜLLEN. DIE CROWDSTRIKE-ANGEBOTE UND CROWDSTRIKE-TOOLS SIND NICHT FEHLERTOLERANT UND NICHT FÜR DEN EINSATZ IN GEFÄHRLICHEN UMGEBUNGEN AUSGELEGT ODER VORGESEHEN, DIE EINE AUSFALLSICHERE LEISTUNG ODER EINEN AUSFALLSICHEREN BETRIEB ERFORDERN. WEDER DIE ANGEBOTE NOCH DIE CROWDSTRIKE-TOOLS SIND FÜR DEN BETRIEB VON FLUGZEUGNAVIGATION, NUKLEARANLAGEN, KOMMUNIKATIONSSYSTEMEN, WAFFENSYSTEMEN, DIREKTEN ODER INDIREKTEN LEBENSERHALTENDEN SYSTEMEN, FLUGVERKEHRSKONTROLLE ODER ANWENDUNGEN ODER ANLAGEN BESTIMMT, BEI DENEN EIN AUSFALL ZU TOD, SCHWEREN KÖRPERVERLETZUNGEN ODER SACHSCHÄDEN FÜHREN KÖNNTE. Der Kunde stimmt zu, dass es in der Verantwortung des Kunden liegt, die sichere Nutzung eines CrowdStrike-Angebots und der CrowdStrike-Tools in solchen Anwendungen und Installationen zu gewährleisten. CROWDSTRIKE ÜBERNIMMT KEINE GARANTIE FÜR PRODUKTE ODER LEISTUNGEN VON DRITTANBIETERN.
— https://www.crowdstrike.com/terms-and-conditions-de/

I guess all these All-Deciding-Managers never read such.

I think about a nice song of Manfred Manns Earth Band: We don't guarantee… which is playing in my brain now

Catweazle

@DoctorG, we 'll see, the affected companies will surely demand compensation and they don't care who pays it, whether it is M$ who pays for the damage or this supposed security company. One thing in any case is clear, M$ has infinitely more resources than this Crowd thing and it is surely not going to pay for this company's mistake (25,000 users it has). I have a strong suspicion that they are going to go to hell, with or without this absurd TOS of theirs. Whoever breaks it pays, it's that simple.

Pathduck

I don't generally watch talking-heads videos but I found this one to the point and well worth watching to understand the technicalities of how this could happen:

CrowdStrike IT Outage Explained by a Windows Developer

Catweazle

@Pathduck, nice BSOD shirt

paul1149

I have precious little regard for either Crowdstrike or Windows.

I do have Win 10 in a virtual machine on my Linux install here, for the sake of one program I spend a lot of time on, but I disconnected the virtual machine from the network adapter, so Windows cannot update and has a high degree of security and privacy. That leaves me in control, and that's how I like it.

Catweazle

@paul1149, it's not a Windows problem, it's a problem of a security sof used by corporations which works from the kernel of the OS. If this soft has a bug creating an conflict, the system crash, yes or yes, irrelevant if it is Windows, Linux or Mac.
See the video in the message from @Pathduck, there they explain it very well.

wintercoast

CrowdStrike published a Post Incident Review this morning. The details are more involved than one might be led to believe from the comments in this thread.

Also, although Linux has not been impacted this time, see CrowdStrike Also Broke Linux Some Months Ago!

wintercoast

@wintercoast They've published a Root Cause Analysis now.

TLDR

"What Happened

The CrowdStrike Falcon sensor delivers AI and machine learning to protect customer systems by identifying and remediating the latest advanced threats. In February 2024, CrowdStrike introduced a new sensor capability to enable visibility into possible novel attack techniques that may abuse certain Windows mechanisms. This capability pre-defined a set of fields for Rapid Response Content to gather data. As outlined in the RCA, this new sensor capability was developed and tested according to our standard software development processes.

On March 5, 2024, following a successful stress test, the first Rapid Response Content for Channel File 291 was released to production as part of a content configuration update, with three additional Rapid Response updates deployed between April 8, 2024 and April 24, 2024. These performed as expected in production.

On July 19, 2024, a Rapid Response Content update was delivered to certain Windows hosts, evolving the new capability first released in February 2024. The sensor expected 20 input fields, while the update provided 21 input fields. In this instance, the mismatch resulted in an out-of-bounds memory read, causing a system crash. Our analysis, together with a third-party review, confirmed this bug is not exploitable by a threat actor.

While this scenario with Channel File 291 is now incapable of recurring, it informs the process improvements and mitigation steps that CrowdStrike is deploying to ensure further enhanced resilience."