What went wrong with Crowdstrike on July 19? The security specialist shares an initial analysis of how it crashed millions of Windows PCs.

July 19 was a pitch-black day for Crowdstrike. A bug in the Sensor Tower software crashed 8.5 million PCs and servers worldwide, causing unprecedented chaos. We already knew that a faulty update was the cause of the problems, and Crowdstrike is now sharing more information about how this update passed the testing process.

In a (preliminary) post-incident analysis, Crowdstrike describes what went wrong on that damned July 19th and how. First, it’s helpful to know that Crowdstrike applies two types of updates: Sensor contentUpdates and Rapid response content-Update. Last Friday’s update was a Rapid response contentUpdate to keep your security software up to date with new types of threats.

What went wrong?

Crowdstrike introduced a new “template type” in February, code with predefined fields on which instances are created to deliver these rapid response content updates. The template type was tested and released in February and March, and was used once before in April to roll out an update for Windows, with no significant issues.

Then we arrive in July. For the update of July 19, two new instances were set up, which were again given the green light by the Content validatorCrowdstrike offers little explanation as to what exactly the Content Validator is supposed to do, but based on the name, we assume this system is designed to verify the content of updates. Apparently, the system went down on July 19, allowing the malicious update to run its course.

warning

To cut a long story short, Crowdstrike relied (too) blindly on its automated testing process and is now paying a high price for it. However, the company previously received a warning when a sensor update caused problems on Linux.

Going forward, it promises to test and validate updates more thoroughly before rolling them out. Customers will also have more control over when updates are performed. This may result in customers updating late, but at least malicious updates will not be installed automatically.

These are Crowdstrike’s first conditional conclusions. The security company promises to publish a detailed analysis later.

European rules

Microsoft has now completed its own analysis. The software giant is clearly pointing the finger of blame at Europe. It is rare that a bug in third-party software can cause Windows to come to a complete standstill. According to Microsoft, the fact that this has now happened is the result of an agreement with the European Commission from 2009 that requires Microsoft to grant parties such as Crowdstrike access to the core of the Windows operating system.

Most of the affected systems are now back up and running, but the solution is not that simple. Is your company still recovering from the Crowdstrike crash? You can visit the Crowdstrike recovery portal or follow our workaround.