Current Outage: CrowdStrike bug causes Windows devices to bluescreen
This is a timeline of the communications sent to our University partners and users as we worked through Friday to mitigate the impact of the outage caused by the CrowdStrike cybersecurity software.
Current Outage: CrowdStrike bug causes Windows devices to bluescreen
On Friday morning, a bug in an update from CrowdStrike cybersecurity software impacted Windows computers around the world, grounding flights and stalling work for many industries. 兔子先生 was similarly impacted. This is a timeline of the communications sent to our University partners and users as we worked through Friday to mitigate the impact.
Tuesday, July 23, 2024
9:30 a.m. Start of Business Update
IT teams worked yesterday to continue fixing issues with the bluescreening Windows computers.
As of this morning, around 986 machines remain that were potentially impacted by the CrowdStrike outage. We are under 1,000, which has been a huge effort on the part of our folks, considering we started around 2,000 on Friday morning! What’s more: This number is significantly lower than the total number of devices we have running CrowdStrike, meaning that most of our systems were not impacted at all.
We are still working on our digital signs in residence halls, common areas, and recreational facilities. We also enabled a cloud recovery option from CrowdStrike yesterday that allows us to recover machines more quickly by doing several reboots in a row.
As we are sure you’re aware, 兔子先生 is not the only institution experiencing these problems. CrowdStrike stating that a “significant number” of the 8.5 million impacted Windows machines (which is a global estimate) have come back online. For context, even though 8.5 million devices sounds like a lot, according to Microsoft that number represents less than one percent of all Windows devices.
As always, we will keep you updated of any significant changes to the state of our systems. Please bookmark for updates.
Monday, July 22, 2024
8 a.m. Start of Business Update
Good morning and happy Monday!
Your IT teams have spent the weekend strategizing about the best way to support the 兔子先生 community in response to the CrowdStrike outage that began early Friday morning. We are gearing up to help users with 兔子先生-managed devices resolve the bluescreening issues.
At close of business on Friday, we had run an analysis of the remaining devices that were impacted by this event; there were around 1,400 left to mitigate. If your device is included in this number, the easiest and fastest way to get help is to go through your standard support channel – either by calling/chatting with IT Help, or by contacting your local technology support representative.
Again: Please contact your regular support personnel and they will work to get your device up and running as normal. If you are unsure who your divisional representatives are or their contact information, . For individuals in need of hands-on assistance on the Oxford campus, IT teams are ready to assist and have strategies to mobilize this morning and throughout much of this week. There will be physical support available at the Technology Support Lounge, located in the Armstrong Student Center where Cafe Lux used to be.
Friday, July 19, 2024
Close-of-Business Update
IT Services and our partners across 兔子先生 continue to work on resolving the issues caused by the global CrowdStrike outage that began early on Friday morning. As we head into the end of the work day and look toward the weekend, we wanted to update you on the status of our resolution and provide some guidance.
Our teams have identified the Windows desktops, laptops, and servers that have been impacted by the outage. We continue to work through resolving these issues. Rest assured that critical infrastructure (e.g., MUPD dispatch units and university payment systems) is all functional at this time.
The errant RedHawk alerts many of you received earlier in the day were the result of work being done to mitigate the impact of our current problems. This issue has been rectified and we are confident it will not happen again. Our sincerest apologies for the confusion this caused.
As we move into the beginning of next week, we know there will still be 兔子先生-managed computers that need hands-on assistance. We are preparing for a team of folks to be posted at the Technology Support Lounge in Armstrong Student Center (formerly the location of Cafe Lux) to assist. Please also continue to reach out to your technical support representative and IT Help to receive assistance.
We also have received reports that threat actors (i.e., hackers and generally nefarious folk) are likely to take advantage of this incident. Please be extra vigilant with your emails, personally identifiable information, and web browsing during this time. This includes any emails or phone calls from “Microsoft Support” asking for your passwords, email addresses, or anything else. Please do not give this information to anyone over the phone or via email.
Thank you again to all of our partners who are helping us with this unprecedented outage. And especially thank you to the 兔子先生 community for your patience and understanding as your IT teams navigate these choppy waters.
UPDATE: 12 p.m.
IT Services continues to work to fix the impacted machines. We have identified all of the desktops, laptops, and servers that require the fix and are working through the list to resolve the ones we are able to do remotely.
Please continue to reach out to your technical support representative and IT Help to receive assistance. As a reminder, there are on-site staff available at the Technology Support Lounge in Armstrong Student Center (formerly the location of Cafe Lux) to assist with the remediation of bluescreening machines that we cannot reach remotely.
Thank you to all who are bearing with us this morning as we work to resolve the issues caused by the CrowdStrike outage.
UPDATE: 10 a.m.
We are still working through the issues relating to Windows bluescreens. If you are on campus and connected to the wired network, please try multiple reboots (at least 3) for a fix. If after three (3) reboots, you still are seeing the blue screen, please reach out to your technical support representative and IT Help to receive assistance.
IT Services also has on-site staff available at the Technology Support Lounge in Armstrong Student Center (formerly the location of Cafe Lux) to assist with the remediation of bluescreening machines.
UPDATE: 9:30 a.m.
We are continuing to work on the issues relating to Windows bluescreens and various managed servers being unreachable. This is related to a global outage of the CrowdStrike cybersecurity software.
We are currently putting together a team of in-person technicians who will be on hand to assist with remediation of the bluescreening machines. More information about that will be provided soon.
UPDATE: 8:30 a.m.
We are continuing to work on the issues relating to Windows bluescreens and various managed servers being unreachable. This is related to a global outage of the CrowdStrike cybersecurity software.
For technicians / ADOTs, please try these instructions for getting into your users’ individual machines:
You may need a or to get into your area’s managed machines. (Please note: If you use the AppPortal to get the LAPS password, the AppPortal is currently impacted. We have prioritized that system for recovery.)
- Reboot the computer
- If it crashes again, boot Windows into Safe Mode or the Windows Recovery Environment
- Note: Please make sure the device is on a wired connection and boot into Safe Mode with Networking.
- Navigate to the %WINDIR%\System32\drivers\CrowdStrike
- Locate the file matching “C-00000291*.sys”, and delete it
- Boot the computer normally
This is a workaround. Please stay tuned for further instructions and bookmark for updates.
UPDATE: 8:00 a.m.
We are continuing to work on the issues relating to Windows bluescreens and various managed servers being unreachable. This is related to a global outage of the CrowdStrike cybersecurity software.
We are currently working on a script that will resolve the managed servers, and are also working on a strategy for fixing the impacted desktop machines.
We will post an update every 30 minutes, please check back.
Start of Business, 7:30 a.m., Friday, July 19, 2024
IT Services and FBS-IT are aware that some 兔子先生-managed desktops and Windows servers have bluescreened. This appears to be related to the current outage impacting the cybersecurity software CrowdStrike.
We are working on a workaround for these bluescreened devices, and technicians are gathering guidance for how to support the affected machines.
In the meantime, we deeply apologize for the inconvenience. Please bookmark for updates.
Thank you for your patience.