According to the 2020 Global Threat Intelligence Report, “cyber-attack volumes increased across all industries between 2018 and 2019.” Web shells, exploit kits and targeted ransomware are just a few tools that bad actors use for attacks. Mostly, such attacks are still successful due to organizational practices related to networks, operating systems and application configurations, testing, security controls and overall security posture. Attackers are still trying to exploit vulnerabilities which are several years old and have patches available, but nevertheless are not being addressed by many organizations’ patch and configuration management programs.
Persistent exploitation of old and famous vulnerabilities such as HeartBleed (CVE-2014-0160) make OpenSSL the second most targeted software technology involved in 19% of hostile activity globally. According to researchers, OpenSSL was the most targeted technology in both the manufacturing and technology industries.
OpenSSL and Glibc Patching Best Practices
Geographically, OpenSSL was the second most attacked technology in Australia and it was the 14th most attacked in Japan. It has had patches available for over two years, but still bad actors are trying to exploit it…and they still succeed: NTT researchers identified that attacks on OpenSSL accounted for 71% of vulnerabilities targeted in the technology industry. All other targeted vulnerabilities accounted for about 1% of hostile activity.
Due to the popularity of OpenSSL vulnerabilities as an attack surface, organizations in all industries must ensure they are promptly mitigating those vulnerabilities to prevent exploitation. Several security frameworks contain recommendations to help organizations mitigate risks.
The MITRE ATT&CK framework, the most holistic framework many organizations adopt to fight cybersecurity threats, recommends organizations to regularly “scan externally facing systems for vulnerabilities and establish procedures to rapidly patch systems when critical vulnerabilities are discovered through scanning and through public disclosure.”
Rebootless Security Updates
Most security updates are applied on reboot (for kernel security updates) or services restart (for shared libraries and other components’ updates). Where there is a reboot or restart, there is downtime, mitigation delays and service interruptions. Most companies tend to plan reboot cycles or maintenance windows to apply patches during off-peak hours of work.
Establishment and ongoing management of patching cycles costs a lot: the average enterprise is spending $100,000 or more per year to organize and manage reboot cycles and maintenance windows to patch their systems. However, reboots in most enterprises are pre-scheduled, meaning vulnerabilities can go unmitigated for months.
Challenges of Libraries Patching
Updating memory-resident shared libraries requires restarting services or rebooting servers. Not all services can easily be stopped. In the Glibc case, restarting services is often similar to rebooting servers with all the same implications and increased risk on the business. In both cases, kernel updates and library updates, downtime and patching delays are now avoidable.
There is another challenge with scanning and identifying the gaps in security when it comes to OpenSSL or glibc scans. It is often difficult to determine which services need to be restarted and which do not. Enterprise servers sometimes have 1000+ active services – which ones use which libraries? Up until now, admins have had very little automated assistance in sorting this out: no vulnerability scanner will detect outdated libraries in-memory. They only check that the libraries on disk are the latest versions.
That being said, as a sysadmin, you may never know if an active service is using an outdated shared library in-memory. Even with the latest Glibc or OpenSSL update on disk, without a restart active services can still be using vulnerable library versions in memory. This is the security gap bad actors often exploit.
How KernelCare+ Addresses Library Patching Challenges
To address these challenges, we designed KernelCare+. In addition to rebootless Kernel patching, it detects all vulnerable shared libraries in-memory and automatically applies live security updates to them without requiring service restarts.
On a high level, it works as follows:
1. The Patch is Created
A library’s source code – both original and patched – are translated into assembly language. These files are compared and new code is put in the different sections of the same Executable Linkable Format (ELF) file. After the code is compiled and linked the patch is extracted from the resulting binaries. The patch files are extracted from the ELF sections.
2. The Patch is Uploaded
The binary files are combined as a single patch, which is then uploaded to a dedicated KernelCare+ patch server. The patch server is used to distribute the patch to customer servers. The patch server can be in the cloud or secured within the customer infrastructure.
3. The Patch is Downloaded
An agent program on each server, lcarectl, ‘talks to’ the patch server, which looks for known libraries on the server. The agent program then downloads the patch needed for each library present on the server.
4. The Patch is Applied
Using Linux APIs, memory near a library is allocated, and the patch is copied into it. After ensuring that no threads are executing the old library code, the agent program reroutes calls from old code to the new patched versions via unconditional jumps.
Once this process is complete, the server’s libraries are fully protected against all known attacks.
Right now, KernelCare+ patches the Glibc and OpenSSL libraries, because these are the ones most often attacked. In the future, it will patch more shared libraries, such as those related to PHP and Python.
You can try KernelCare+ for 30 days on all your servers here. No credit card required. It is now available for RHEL 7, CloudLinux OS 7 and CentOS 7, but more distributions will be added in the following weeks.
Organizations need to deliver consistent, available, secure and reliable services to prosper. Spending resources on the prevention of cyber-attacks is always preferable to spending resources on recovering from them. History has shown that some organizations never recover from major security incidents. Given the frequency of successful exploits against shared library vulnerabilities, organizations need to adopt and deploy live patching for both Linux kernels and shared libraries.
