Global IT outages disrupt key sectors globally due to updates

Multiple businesses across the globe faced significant disruptions due to the extensive IT outages linked to crucial software updates from renowned industry giants Microsoft and CrowdStrike.

This event reverberated through critical sectors such as aviation, media, banking, and healthcare, with far-reaching impacts noted in the US, Australia, Germany, and the UK.

Sascha Giese, Global Tech Evangelist at SolarWinds, offers a broad perspective on the complexities behind such widespread issues. "What we saw this morning is the result of an update that came with unexpected side effects. It would be easy to say it's a QA fail, but the reality is a little more complex. Security vendors are under constant pressure to keep up with zero-days, new ransomware, dynamic botnets, and nowadays even AI-driven threats. They have to be on their toes when it comes to product development and updates," Giese elaborated.

The automatic nature of these updates is a necessity given the dynamic threat landscape. However, there is often insufficient time to thoroughly test each update, compounded by a lack of manpower within organisations for extensive testing. Giese emphasised the importance of assessing how vendors handle such situations: "The priority now is to see how the vendor handles the communication, the fallout, and the fix."

CrowdStrike, a major cybersecurity player, was particularly hard hit, with their EDR agent identified as the source of errors leading to these significant disruptions. Kevin Reed, Chief Information Security Officer at Acronis, commented on the situation: "The recent CrowdStrike outage appears to stem from a bug in their EDR agent, which was unfortunately not thoroughly tested. This resulted in widespread disruption as many installations were affected globally." The affected systems require manual rebooting in "safe mode" and deletion of a faulty driver file, a cumbersome process that temporarily leaves systems vulnerable to opportunistic attacks.

Reed further highlighted the importance of rigorous testing and staged updates, and questioned the self-protection mechanisms of CrowdStrike's software, noting, "This issue reminds us how fragile IT infrastructure is and why cybersecurity should be natively integrated with backup. An integrated solution is the only way to provide complete protection that would enable fast roll-back to the working state." Additionally, Reed advised businesses to have robust backup solutions to mitigate such risks and ensure minimal downtime.

In the UK, the NHS and numerous businesses experienced disruptions, prompting insights from Keiron Holyome, VP UK and Emerging Markets at BlackBerry Cybersecurity. Holyome stressed the need for a proactive cybersecurity strategy and robust event management systems. "Given this outage is impacting some of the most critical systems, networks and applications in the world, the response must be met with speed, accuracy, and accountability. Here, a critical event management (CEM) solution can provide real-time visibility to ensure a quick and informed response as the crisis evolves," Holyome stated.

He also touched upon the broader implications, reflecting on the outdated nature of legacy cybersecurity practices. "This is likely another example of legacy cybersecurity practices in play, with complex EDR and heavy endpoint agents a major infrastructure risk and unnecessarily complex. Using a lightweight AI on the endpoint can avoid these types of outages, as it protects your environment without heavy agents and regular updates that put your operations at risk," he explained.

Holyome concluded by underscoring the necessity of continuous vulnerability and risk assessment through regular testing. "Today's global IT outage serves as a stark reminder that the best defence is a good offence. Understanding your vulnerabilities and risks through regular testing is paramount," he added.

The recent IT outages highlight vulnerabilities within even the most established systems, underscoring the need for continuous evaluation and improvement in cybersecurity measures across all sectors. The responses and remedial measures taken by these companies over the next few days will be crucial in restoring not only their systems but also user trust.