Software updates play a critical role in protecting systems from cyber threats as well as providing new and improved functionality to software products.
They are necessary to patch vulnerabilities that can be exploited by malicious actors, ensuring that systems remain secure.
Software updates are one of the four pillars of the 2024 international Cybersecurity Awareness Month campaign.
As part of the campaign, authority organizations, such as the US Cybersecurity and Infrastructure Security Agency (CISA) and the National Cybersecurity Alliance issued a list of update-related recommendations relating to software updates:
- Verify the source of the updates you are notified of
- Apply updates as soon as they are available
- Turn on automatic updates when available
These recommendations are good practices, especially for the general public when thinking about their connected devices.
However, for large organizations software update management can be complex.
Updating software can introduce new business risks, as recent incidents have demonstrated.
In July 2024, a faulty update of the CrowdStrike Falcon sensor crashed roughly 8.5m Windows computers, forcing them to display Microsoft’s blue screen of death (BSOD).
Infosecurity spoke with several experts to explore the role of software updates in cybersecurity and how to prevent them from being the cause of security disasters.
Understanding the Role of Software Updates
Unpacking the Patch Management Taxology
In patch and vulnerability management, developers, IT and cybersecurity professionals use many different terms, including update, upgrade, rollback, patch, hotfix and bugfix.
An upgrade is a major software version change that can sometimes involve hardware replacement.
An update is a less significant version change that can generally be pushed in a few minutes.
A rollback occurs when software users or owner needs to remove the latest version and install a previous one for operational or security reasons or because the newest version was faulty.
An update can sometimes include a patch, a set of modifications to the software that can include bug fixes, the correction of an error or defect in software code, alongside other changes, such as security updates, performance improvements, or new features.
A hotfix is a type of small patch specifically designed to address a critical issue that needs to be resolved immediately. It is generally pushed in real-time and typically used to fix urgent bugs, security vulnerabilities, or other problems that can't wait for the next scheduled release.
Patches that modify the kernel or core system files almost always require a reboot. Patches that modify specific applications or services may or may not require a reboot, depending on the nature of the changes.
Why Updates are Critical to Software Security
Today, the most common method attackers use to obtain initial access to organizations is by exploiting known software vulnerabilities.
According to Mandiant’s M-Trends report, published in April 2024, attackers exploited vulnerabilities to gain initial access in 38% of intrusions in 2023, a 6% increase from the previous year.
Security vulnerabilities in software are typically exploited within 19 days of being discovered, and yet, organizations take over 100 days on average to install updates that fix these vulnerabilities, according to Skybox Security findings.
Updates and associated patches are the only way to perennially fix vulnerabilities, this is why they are crucial in every organization’s cybersecurity posture.
Why Updates Can Be a Security Challenge
Software updates can introduce operational issues. For instance, Rose Gupta, threat and vulnerability management lead at AssuredPartners, shared her experience with an August 2024 patch on Windows Server 2019 that caused disk issues and made servers unresponsive for AssuredPartners and its customers.
“Microsoft told us to hold off until the September patch, but since there were a lot of zero-day vulnerabilities patched in the August update, we still needed to find mitigation measures,” she explained.
Software updates can also pose security risks to major security providers. In September, an Apple macOS 15 ‘Sequoia’ update disrupted security kit including CrowdStrike, SentinelOne and Microsoft tools.
Cybercriminals can also infect a software update in order to infiltrate an orgnaization and infect target devices.
This technique was deployed for good in the takedown of Ghost, a dedicated encrypted communication platform used by cybercriminals. The Australian Federal Police infiltrated the crimeware app by modifying an app update.
Software Update Best Practices
In the case of the CrowdStrike-induced IT outage, the responsibility has largely been attributed to CrowdStrike itself, with VP for counter-adversary operations Adam Meyers apologizing before the US Congress, and, to a lesser extent, Microsoft.
According to Josh Chessman, an advisor at Lionfish Tech Advisors, the CrowdStrike leadership was “arrogant” for pushing the same update to all its users at once, and the fact that its Falcon sensor had Microsoft kernel access made the impact much worse.
He also said that Microsoft should have guardrails in place to avoid such a broad-scale incident.
However, he stated that the CrowdStrike customers most impacted by the outage, including Delta Airlines, likely lacked the best patching and update practices.
Read more: CrowdStrike Windows Outage - What We Can Learn
Build a Comprehensive Cartography of Your Assets
AssuredPartners’ Gupta said the first stage of a good update and patch management process is identifying all your assets.
However, Chessman added that an inventory is generally insufficient, especially for larger organizations. “They would have a multitude of operating systems, sometimes many different versions of the same, as well as a range of software and applications, also in various versions, each managed by a different team,” he explained.
He highlighted the need to map your assets and know in which business units they are rather than merely list them.
“Importantly, it’s preferable not to do this on a spreadsheet but in an easily updatable, machine-readable format,” he added.
Endpoint management solutions can help.
Additionally, Gupta said that building a software bill of materials (SBOM), which lists each software package and their dependencies, can be of real value in patch management.
Read more: Navigating the Vulnerability Maze: Understanding CVE, CWE, and CVSS
Develop a Responsibility Assignment Matrix
Gupta also argued that organizations should develop a responsibility assignment matrix (RAM), also known as the responsible, accountable, consulted and informed (RACI) model.
This helps the security teams know who is responsible for which types of patches and assign them quickly if needed.
“At AssuredPartners, this allows us to avoid wasting time trying to find the correct owners, which initially took up much of our time,” she continued.
Managed Automated Updates, Not Automatic Updates
While turning on automatic updates can be a rule of thumb for individuals, Chessman says it is a more complex issue for organizations.
On the one hand, organizations with a strong need for network availability (e.g. manufacturing companies) would want to avoid disrupting systems without prearrangement. On the other, fully automatic updates can have dire consequences, as the CrowdStrike incident showed.
“There is likely not someone at Delta Airlines or at any other impacted organization who decided at 4 am to push the CrowdStrike update, but there is someone who decided to automatically push any update coming from the CrowdStrike Falcon sensor,” Chessman explains.
Meanwhile, organizations cannot push each update manually. “The larger the business, the greater the multitude of different devices and software applications, the more you need to automate,” he continued.
This is why the advisor recommended adopting a managed automated software update process rather than a fully automatic update stance. “Use dedicated software if you can afford them,” he added.
Dedicate an Automated Testing Environment
Gupta and Chessman highlighted the need for a testing environment in which updates can be pushed first before applying them to the whole business.
AssuredPartners has deployed both a user acceptance testing (UAT) lab and a non-production testing group made of endpoints on which updates and patches can be tested first before going into production.
Chessman added that another layer of testing could employ ‘security champions’ or ‘early adopters’ within customer organizations that volunteer to try updates on their network.
Gupta explained to Infosecurity: “Now, I'm building an automated testing process to assess the impact of patches on performance, with metrics like browser response time, CPU usage and network latency.”
Prioritize and Stage Patches
Another lesson Gupta has learned from building a vulnerability management program is the need to prioritize.
“When identifying a vulnerability, we will apply patches depending on several criteria. For this, we’ve built our own model, which is very similar to the Vulnrichment program, run by the US Cybersecurity and Infrastructure Security Agency (CISA) but curated for AssuredPartners,” she said.
This model takes into account the intrinsic severity of the vulnerability (CVSS score), the exploitability likelihood (EPSS score), the criticality of the vulnerability to AssuredPartners and business and financial impact.
“We use the Stakeholder-Specific Vulnerability Categorization (SSVC) framework to prioritize. If critical, we can run tests for 24 hours and then roll out the patch in production,” Gupta explained.
Chessman added that organizations should stage their patch deployments based on the defined priorities and develop a multi-stage approach that includes testing labs and testing groups.
"CrowdStrike knew within 45 minutes that things were going badly and stopped sending the update. The issue is that they had already ruined millions of computers at that point. With a staged approach, they could have mitigated the impact. The same goes with their customers,” Chessman argued.
One of CrowdStrike’s post-outage measures was to deploy a staged approach to rapid response content updates.
Read more: How to Disclose, Report and Patch a Software Vulnerability
Select Adequate Update Timings
The massive impact of the CrowdStrike-induced IT outage was partly due to the nature of some of those affected being major airlines combined with the timing, as most systems went down on Friday, one of the busiest days of the week for transport companies.
A typical social media criticism from the cybersecurity community pointed to the timing chosen by CrowdStrike for pushing this update – late on Thursday.
According to Chessman, Thursday or Friday is not necessarily a wrong time to push updates, but it depends on your business.
“For some Monday to Friday organizations, updating on Friday might make sense so that if they experience an issue, they have the weekend to recover from it. For others, no one works on the weekend, and they’ll want to avoid Friday patches,” he explained.
Have a Contingency Plan and Build Resilience
For Chessman, one of the main lessons learned from the CrowdStrike incident was the importance of having a contingency plan.
“Assume everything is going to hell and have a plan for when that happens, on how you will work around the issue and recover from it,” he said.
Additionally, he advised organizations who can afford it to use different, competing solutions when they can.
“It is costly, not always practical or even realistic, but if some CrowdStrike customers had a competing solution installed on some of their endpoints, the outage would surely have had a lesser impact,” he added.
Train People on the Most Critical Systems and Software
Finally, Chessman highlighted the general need for more training on tools that IT and security teams use daily, especially security tools.
“One thing I see regularly is that organizations buy security tools but do not provide the people using and administrating them proper training, teaching them what all the functionalities are and the performance and security implications associated with using them,” he concluded.
Conclusion
Software updates are essential for maintaining system security and providing new functionality to systems, but performing software updates also introduce potential risks. As demonstrated by recent incidents, even well-intentioned updates can lead to unintended consequences.
To mitigate these risks, organizations must implement robust update management processes that balance the need for security with the potential for disruption.
By following best practices such as verifying update sources, applying updates promptly, and enabling automatic updates when appropriate, organizations can significantly reduce the likelihood of update-related security incidents.
Additionally, investing in comprehensive testing and quality assurance can help identify and address potential issues before they impact production systems.
Ultimately, a proactive and informed approach to software updates is crucial for ensuring the security and resilience of modern IT environments.