What went wrong with Crowdstrike on July 19? The security specialist shares an initial analysis of how it crashed millions of Windows PCs.

July 19 was a pitch-black day for Crowdstrike. A bug in the Sensor Tower software crashed 8.5 million PCs and servers worldwide, causing unprecedented chaos. We already knew that a faulty update was the cause of the problems, and Crowdstrike has released more information over the past few days about how that update passed the testing procedure. Following a preliminary report a few days after the crash, here is the final report.

In the post-incident analysis, Crowdstrike describes what went wrong on that damned July 19th and how. First, it’s helpful to know that Crowdstrike applies two types of updates: Sensor contentUpdates and Rapid response contentupdate. The update from July 19 was a Rapid response contentUpdate to keep your security software up to date with new types of threats.

What went wrong?

Crowdstrike introduced a new “template type” in February, code with predefined fields on which instances are created to deliver these rapid response content updates. The template type was tested and released in February and March, and was used once before in April to roll out an update for Windows, with no significant issues.

Then we arrive in July. For the update of July 19, two new instances were set up, which were again given the green light by the Content validatora system that validates the content of updates. A discrepancy between the number of input fields the update contained (20) and the number of input fields the content validator should validate (21) caused the system to crash.

warning

To cut a long story short, Crowdstrike relied (too) blindly on its automated testing process and is now paying a high price for it. However, the company previously received a warning when a sensor update caused problems on Linux.

Going forward, it promises to test and validate updates more thoroughly before rolling them out. Crowdstrike is having an independent analysis of the Falcon sensor code done. Customers will also have more control over when updates are executed. This may lead to customers updating too late, but at least malicious updates will not be installed automatically.

European rules

Microsoft has now completed its own analysis. The software giant cites European rules as the cause of the crash. It is rare that a bug in third-party software can cause Windows to come to a complete standstill. According to Microsoft, the fact that this has now happened is the result of an agreement with the European Commission from 2009, which requires Microsoft to grant parties such as Crowdstrike access to the core of the Windows operating system.

According to Crowdstrike, 99 percent of all affected sensors are now back online. Several thousand organizations are still experiencing problems. You can go to Crowdstrike’s recovery portal or follow our workaround.

This article originally appeared on July 24. The text has been updated with the latest information.