Traditional malware detection and forensic investigation techniques typically focus on detecting malicious native executables to disk, and performing disk forensics to uncover evidence of historical actions on a system.
In response, many threat actors have shifted their offensive techniques to avoid writing to disk, staying resident only in memory. Consequently, the ability to effectively analyze live memory for evidence of compromise and to gather additional forensic evidence has become increasingly important.
API hooking is one of the memory-resident techniques cyber-criminals are increasingly using. The process involves intercepting function calls in order to monitor and/or change the information passing back and forth between them. There are many reasons, both legitimate and malicious, why using this might be desirable. In the case of malware, the API hooking process is commonly considered to be ‘rootkit’ functionality and is mostly used to hide evidence of its presence on the system from other processes, and to spy on sensitive data.
How are the cyber-criminals using API hooking?
There are two common use cases for the malicious use of API hooking. Firstly, it can be used to spy on sensitive information and so they use it to intercept sensitive data, such as communications with the keyboard to log keystrokes including passwords that are typed by a user, or sensitive network communications before they are transmitted. This includes the ability to intercept data encrypted using protocols such as Transport Layer Security (TLS) prior to the point at which they are protected, in order to capture passwords and other sensitive data before it is transmitted.
Secondly, they modify the results returned from certain API calls in order to hide the presence of their malware. This commonly may involve file-system or registry related API calls to remove entries used by the malware, to hide its presence from other processes. Not only can cyber-criminals implement API hooking in a number of ways, the technique can also be deployed across a wide range of processes on a targeted system.
Tackling malicious API hooking
One way cybersecurity teams can detect the hidden traces of API hooking and other similar techniques is through memory analysis frameworks such as Volatility. Volatility is an open source framework and the de facto standard toolset for performing memory analysis techniques against raw system memory images, useful in forensic investigations and malware analysis.
While memory analysis can be an incredibly powerful and useful technique, it does not come without its challenges. One hurdle to consider when deploying memory analysis is the labor intensity it requires: memory analysis is a highly skilled and time-intensive technique typically performed on one image at a time. This can be very effective when performing a dedicated investigation of a serious compromise, where the systems involved are known and relatively small in number.
However, the challenge arises when trying to use memory analysis at scale to detect compromises on a large enterprise network in the absence of any other evidence.
Another obstacle to be aware of when implementing memory analysis is legitimate ‘bad’ behavior. There are plenty of examples of hooking techniques being used by malware for malicious purposes. Nevertheless, there are also many cases of these techniques being used for legitimate, above-board purposes. In particular, technologies such as data loss prevention and anti-virus often target the same functions for hooking as malware does. Without the techniques and experience to quickly separate legitimate injection and hooking from malicious behavior, a great deal of time can be wasted.
Successful attack detection and response
As a first step in dealing with techniques like this, organizations need the capability to easily retrieve system memory images from suspicious machines to allow rapid response and aid forensic investigation. However, this can generally only be used in a reactive manner.
To perform effective attack detection and response at scale - specifically with regard to these techniques - an ability to conduct memory analysis proactively at scale across an enterprise network is required, which is where toolsets continuously conducting live memory analysis and reporting on suspicious findings are required. This will enable the proactive discovery of unknown memory-resident malware without any prior knowledge or signatures.
Good endpoint detection and response software, which offer live memory analysis capabilities at scale, are required to proactively detect the direct use of techniques such as live memory analysis.
Additionally, when gathering results at scale, approaches such as anomaly detection can help greatly by drawing a dividing line between API hooking that is common across the network, probably due to security software in use, and anomalous API hooking that seems present only in a few isolated cases. Traditional memory forensics using a tool such as Volatility can then be used in order to investigate, in detail, systems exhibiting suspicious behavior.
Many malware families have moved to using techniques such as API hooking in a stealthy attempt to avoid traditional security solutions and achieve certain end goals, such as spying on passwords. The 2015 Verizon Data Breach Report found that “malware is part of the event chain in virtually every security incident”. It also reported that “70-90% of malware samples are unique to an organization” and that “organizations would need access to all threat intelligence indicators in order for the information to be helpful”.
Given these findings, it is obvious that having an effective technique for discovering previously unseen malware on your network is extremely important. Memory analysis can be used to uncover some, not all, of the stealth techniques used by modern malware families but it is an important capability to have in order to detect compromises using modern memory-resident malware.