Windows blue screen of death (BSOD) issue after CrowdStrike update leaves many complaining

Update 26/07/24: CrowdStrike CEO has now confirmed that nearly all affected devices across the globe are now back online as of July 25. Here’s the full statement:

Update 25/07/24 04:49 pm (IST): TechCrunch reports that CrowdStrike is offering a $10 Uber Eats gift card to some as an apology for the massive outage. This gesture, confirmed by CrowdStrike spokesperson Kevin Benacci, was intended for “teammates and partners who have been helping customers through this situation.” However, some recipients reported issues redeeming the voucher, with Uber flagging it as fraud due to high usage rates.

Update 24/07/24 04:49 pm (IST): In a statement to The Wall Street Journal, Microsoft seems to be pointing its fingers at the European Commission for its inability to prevent the situation that unfolded. Microsoft claims that, unlike Apple, it “cannot legally wall off its operating system” due to a 2009 agreement with the European Commission. This agreement requires Microsoft to give third-party security app developers the same level of access to Windows as Microsoft itself has. The company suggests that this commitment prevents it from implementing safeguards similar to those in macOS, which doesn’t grant kernel-level access to third-party apps, thus avoiding the type of catastrophic error that occurred on Windows systems.

Update 23/07/24 04:49 pm (IST): As usual, scammers are trying to take advantage of the situation and trick affected users into paying them to restore affected machines or pretend to be associated with the company. CrowdStrike has listed a number of such websites that must be avoided. Here’s the list for reference:

crowdstrike.phpartners[.]org
crowdstrike0day[.]com
crowdstrikebluescreen[.]com
crowdstrike-bsod[.]com
crowdstrikeupdate[.]com
crowdstrikebsod[.]com
www.crowdstrike0day[.]com
www.fix-crowdstrike-bsod[.]com
crowdstrikeoutage[.]info
www.microsoftcrowdstrike[.]com
crowdstrikeodayl[.]com
crowdstrike[.]buzz
www.crowdstriketoken[.]com
www.crowdstrikefix[.]com
fix-crowdstrike-apocalypse[.]com
microsoftcrowdstrike[.]com
crowdstrikedoomsday[.]com
crowdstrikedown[.]com
whatiscrowdstrike[.]com
crowdstrike-helpdesk[.]com
crowdstrikefix[.]com
fix-crowdstrike-bsod[.]com
crowdstrikedown[.]site
crowdstuck[.]org
crowdfalcon-immed-update[.]com
crowdstriketoken[.]com
crowdstrikeclaim[.]com
crowdstrikeblueteam[.]com
crowdstrikefix[.]zip
crowdstrikereport[.]com

Furthermore, in an exclusive interview with Sky News Australia, the company’s President — Michael Sentonas — did hint that CrowdStrike is prepared to face potential legal consequences and also compensate affected companies for the massive outage. You can watch the complete interview below:

Update 22/07/24 09:54 am (IST): CrowdStrike reports significant progress in restoring affected systems following the July 19 incident that impacted approximately 8.5 million Windows devices. A large number of these devices are now back online and operational. The company has developed a new technique to accelerate system remediation, which is being tested with customers and will be available as an opt-in option. 

Microsoft has also released a new USB Recovery Tool to assist IT administrators in repairing Windows endpoints affected by the recent CrowdStrike Falcon agent issue. This tool automates the manual steps outlined in KB5042421 and can be downloaded from the Microsoft Download Center. The process involves creating a bootable USB drive using a 64-bit Windows client, then using this USB to repair impacted devices. The tool requires administrative privileges, a formatted USB drive with at least 1GB of free space, and BitLocker recovery keys for affected devices. Microsoft provides detailed instructions for generating the USB repair solution and using it on impacted devices. For comprehensive information and step-by-step guidance, you can review Microsoft’s detailed post, which includes links to additional resources and recovery steps.

Update 20/07/24 09:15 am (IST): Apart from the workaround shared below, you can also try to​​​​​​​ roll back to a snapshot before 0409 UTC, as mentioned by CrowdStrike. The company has also posted a “technical overview” of the incident that you can read here. In short, the issue stemmed from a logic error in Channel File 291, which controls how Falcon evaluates named pipe execution. The problematic update was deployed at 04:09 UTC and remediated by 05:27 UTC. This incident was not related to a cyberattack and only impacted Windows systems running Falcon sensor version 7.11 and above that were online during the specified timeframe.

CrowdStrike has since corrected the logic error in Channel File 291 and continues to evaluate and protect against named pipe abuse. The company is conducting a thorough root cause analysis to determine how the logic flaw occurred and to identify potential improvements in their processes. Systems that were not impacted remain protected and face no risk of experiencing this event in the future. Linux and macOS systems were unaffected, as they do not use Channel File 291.

The CEO also apologized for the incident in a follow-up post on X, stating, “We understand the gravity of the situation and are deeply sorry for the inconvenience and disruption. We are working with all impacted customers to ensure that systems are back up and they can deliver the services their customers are counting on.”

Update 19/07/24 5:07 pm (IST): Some users claim they were able to fix the problem by multiple reboots, in some cases, over a dozen times too.

Update 19/07/24 3:44 pm (IST): CrowdStrike’s CEO has issued a statement:

CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts. Mac and Linux hosts are not impacted. This is not a security incident or cyberattack. The issue has been identified, isolated and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website. We further recommend organizations ensure they’re communicating with CrowdStrike representatives through official channels. Our team is fully mobilized to ensure the security and stability of CrowdStrike customers.

Original article published on July 19, 2024 follows:

A widespread issue with CrowdStrike’s latest security update is causing Windows systems to crash, resulting in the dreaded “blue screen of death” (BSOD) for users across multiple countries. The problem, which began on July 18, 2024, is affecting both servers and workstations, leading to significant disruptions for businesses and organizations.

Reports of the issue have flooded social media platforms, with hundreds of users voicing their frustrations on X (formerly Twitter) and Reddit. According to a pinned comment from a CrowdStrike moderator on Reddit, the company is aware of “widespread reports of BSODs on windows hosts, occurring on multiple sensor versions” and is currently investigating the cause, which appears to be an issue with the csagent.sys driver.

Users are getting errors such as “PAGE_FAULT_IN_NON_PAGED_AREA”, “CRITICAL_PROCESS_DIED”, and SYSTEM_THREAD_EXCEPETION_NOT_HANDLED”.

The scope of the problem appears to be extensive, with users reporting outages in the United States, EU, Australia, New Zealand, India, and other regions. We’ve also encountered the issue on our system, which resulted in us discovering the flood of reports online. One user on Reddit claimed their organization, with a fleet of over 50,000 devices, was entirely affected.

Many IT professionals are scrambling to find solutions, with some suggesting workarounds such as renaming the CrowdStrike folder in the Windows system directory after booting into safe mode to mitigate the issue. A mod listed down the steps that you can follow, shared in a tech update on the official status tracker, check them out below:

1. Boot Windows into Safe Mode or the Windows Recovery Environment
2. Navigate to the C:\Windows\System32\drivers\CrowdStrike directory
3. Locate the file matching “C-00000291*.sys”, and delete it.
4. Boot the host normally.

However, these measures are temporary and may not be feasible for all affected systems, especially those with BitLocker protection. The incident has wrecked hundreds of online services, with users taking to services like Downdetector to report issues with various platforms and services. Here’s a screenshot for reference:

The incident has raised concerns about the reliability of security software and the potential risks of having a single point of failure in cybersecurity infrastructure. One Reddit user commented, “Imagine the one thing meant to protect from malware being the malware all along?”

CrowdStrike, a leading provider of cloud-delivered endpoint and workload protection, has yet to provide a comprehensive fix for the issue. While the company has posted a technical alert on their support portal, details about the root cause and a permanent solution are still pending.

The full impact of this event on Crowdstrike’s reputation and the broader implications for the cybersecurity industry remain to be seen.

Earlier reports highlighted a wide scale outage with Microsoft Azure. The problem stemmed from a configuration change in Azure backend workloads, which disrupted connections between storage and compute resources. This interruption cascaded to various Microsoft 365 services, causing user access problems and functionality limitations across platforms such as PowerBI, Microsoft Fabric, Teams, the admin center, Microsoft Purview, and Viva Engage. Some services, including Microsoft Defender, Intune, OneNote, OneDrive for Business, SharePoint Online, and Windows 365, have reportedly recovered.

Microsoft is actively working to mitigate the impact, with some affected users already seeing improvements. The company has prioritized addressing the lingering issues for the remaining degraded Microsoft 365 apps. While some services are in read-only mode, others are experiencing delays in processing events or complete inaccessibility. Microsoft has committed to providing regular updates on the situation and expects to release the next update by July 19, 2024, at 7:30 AM UTC. The tech giant emphasizes that they are treating this event with the highest priority.

Developing…

Featured image source: @troyhunt / X