Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Rootkit detection and removal know-how

Get advice on how to detect malware and rootkits and the best ways to achieve rootkit removal and prevent hacker attacks.

Difficult to detect and nearly impossible to remove, rootkits may already own your systems.

Rootkit is a scary word to a CIO. It conjures visions of worms eating through the network, backdoors opened to sensitive or proprietary information, users unaware of their credit card numbers being stolen, and the stifling cost of incident response. Rootkits are discussed in hushed tones, as if the mere word will summon one from the ether. At the end of the day, rootkits are like any other malware, but tougher to detect and remove. Competitive corporations, organized crime and terrorists are using these tools to attack networks and steal data. While customer data theft can cost a company millions, insider threats are the major problem. More than 70 percent of a company's value may be held in its intellectual property assets, a prime target for competitive intelligence gathering. Rootkits can be used to steal information without detection, which is what makes them so dangerous. Bad guys design rootkits to stay hidden for years, so they have continued access to information. Although they come in many shapes and flavors, suffice it to say, rootkit is a fairly new word for a backdoor. Many techniques used by rootkits were pioneered by virus developers in the early '90s, and the rise of the Internet fueled the need for a remotely accessible backdoor. While Unix systems continue to be targeted, rootkits rapidly evolved to target ubiquitous Windows machines.

Rootkits have become so powerful they can evade desktop firewalls, virus scanners and IDS/IPS products. Today, rootkits are advancing upon cellular phones, the 64-bit Vista operating system and device firmware. Rootkits old and new continue to be a threat to your data.

Hidden Roots
What many IT and security professionals don't know is that modern rootkits are much more powerful and difficult to detect than advertised.

None of the currently available rootkit detection solutions, commercial or research, are effective at detecting rootkits. The failed detection techniques range from signature-based scanning to heuristics. A recent study conducted by SAIC and HBGary for DARPA pitted rootkits, old and new, against commercial and public domain detection tools. Not a single detection tool could detect all the rootkits. Most couldn't even detect more than 25 percent of the sample set. This is startling, considering that many of the sampled rootkits have been in the public domain for several years.

Signature-based virus scanners are severely limited because of polymorphism, a technique that makes every copy of a given malware program unique, each with a different hash. This means that if 3,000 machines are infected, there are 3,000 different hashes for the malware file--yet it remains the same malware with the same capabilities.

If detection cannot even handle rootkits that have been in the public domain for years, how can detection stand a chance against funded adversaries that craft new, unknown rootkits?

Rootkits are evolving far beyond traditional file and process hiding to achieve maximum stealth and evade current commercial security technology. Modern developers design standalone backdoor systems that in-clude the ability to communicate over the network, embedding such complex code as entire TCP/IP stacks that are independent of the operating system.

The difference between traditional open-source rootkits and modern weaponized rootkits is like going to a toy collectors' swap meet versus an invitation-only arms bazaar. Weaponized rootkits are developed with specific bypasses around all the commercial detection solutions. These techniques include anti-forensics and anti-debugging, encryption, covert command-and-control channels, keystroke logging, remote screen video capture, remote audio bugging via microphone, email and pre-encryption/post-decryption text capture, and covert data exfiltration.

There's more. Anti-forensics presents a major problem for detection and evidence collection tools. For example, a major part of forensic analysis is the recovery of file data from the hard drive. If the rootkit or malware doesn't store data on the hard drive, then this form of forensics is useless. In fact, many new rootkits are using "in-memory only" methods specifically to combat disk-based forensics.

The most widely used anti-forensics technique is called "packing," meaning simply that the malware encrypts its data and code, making reverse engineering (see "Putting It in Reverse")--a critical step in analyzing and combating rootkits--far more difficult and costly. It forces your reverse engineers to work through inane software puzzles in order to get at the data underneath. Other anti-forensics techniques include fouling up a debugger or detecting execution within virtual machines.

Burrowing Deep
Regardless of how good your assessment practices are, some rootkits are going to slip by unnoticed and hurt you. This is just the nature of the game.

Enterprise solutions tend to produce too much of the wrong data. They flood you with minor details or information that is not directly pertinent to an attack. Your response staff must wade through all this data to find evidence of real attacks. Filtering solutions throw out actual attack data with the rest of the noise. A data aggregator may collect information from firewalls, HIPS and NIDS, but this information is very high level--source and destination IP addresses, some header information, etc. This may tell you what Web address was used to download a malware binary, but won't store an actual copy of the binary itself. You don't have the data stream--the actual files and packets involved in the attack--that you need to reconstruct an attack.

This data is critical if you are to recover how the attack was able to penetrate the network. Even if the streams are collected, they cannot be stored for very long--a couple of weeks at best--because of storage limitations. Often the streams are lost by the time an incident is detected, perhaps months after the event.

Attackers are well aware of this and design their rootkits to look like noise in the system. For example, stolen data may be transferred out of your network using harmless-looking protocols; what appears to be routine Web or DNS traffic may in fact be a rootkit backdoor in action.

Because new forms of attack are always emerging, automated detection will never fully replace human response teams. Attackers are constantly refining their methods and tools. An effective defense requires human creativity to narrow down potential threats. For example, even if there is no signature to detect a new rootkit, an infected machine may behave in a suspicious way. Attacks are sometimes discovered only after a user in the network notices his or her computer acting strangely. An infection is obvious when machines in the network start port-scanning the network, launching DDoS attacks, or serving pirated music. Some-times an infection is found only after malware or suspicious traffic is captured and analyzed at a network gateway.

Evidence from multiple sources may need to be combined into a big-picture view before threats become apparent. For example, you may cross-reference blacklists of known botnets with outbound network connections or reverse engineered backdoor code.

The work doesn't end once you finally capture a sample of the malware. You can perform an enterprise-wide scan for the malware, but this does not guarantee you will know anything about the nature of the threat. In many cases, machines are found to be infected but nothing will be revealed about how long the malware has been there, how it works, and what data, if any, is being stolen. This requires intense forensic analysis.

Good forensics analysts are difficult to find, so investing in making your existing team more effective makes good business sense. Digital forensics products help make evidence collection easier, but some malware programs employ anti-forensics, making it difficult to recover evidence about how it works or what kinds of information it's stealing.

When dealing with large networks, evidence reasoning can be used to evaluate data from multiple sources (such as network- and host-based IDS) to better reduce false positives. Better tools mean better scalability for your response team.

Remote forensic capabilities are crucial in geographically distributed networks. The response team can make one phone call and have a remote assessment underway: They may have the administrator install an agent on a suspect machine, evaluate it remotely and determine if it needs to be taken offline for further analysis.

When malware is captured, automated reverse engineering can answer many important questions about the unknown malware and can replace hours or days of tedious manual labor.

After a good forensics assessment, your team should have enough information to construct network IDS signatures that can capture variations of the attack at the firewall. Depending on the type of signature you craft, you can capture the malware binary to prevent additional infections or even detect the command-and-control channel, revealing infected locations in the network and curbing their ability to communicate. Email protection can be configured to prevent the binary from being delivered as an attachment or within a download link. Use a host-based intrusion prevention system (HIPS) to scan host systems across your enterprise.

Digging 'Em Out
Extraction and removal is another problem. Like barbed wire in a tree, it can be difficult to extract a rootkit from a system without killing it. For example, a rootkit may not have any normal files on the hard drive. Instead, the rootkit could be inserted into core code of the operating system and boot sequence. Removing the rootkit could leave corruption behind, and the machine will never boot again. Further, you may not be convinced you have completely removed a rootkit. Since rootkits are designed to hide, it's conceivable that you might miss something. Because of these factors, the best choice is to reinstall an infected machine from scratch and not take any chances.

But that's not always feasible, and you may be left with no choice but to attempt to remove the rootkit. The infected machine may be a mission-critical server you cannot take offline. To evaluate a system under these circumstances you can create a drive image. There are many tools that can image a hard drive, producing a raw binary file that can be analyzed without touching the original system. The hard drive image can be rebooted in a virtual machine, such as VMware. Using the virtual machine in conjunction with reverse engineering techniques, such as automatic tracing, a great deal of information about behavior can be captured and reported. You can test attempts to remove the rootkit on the virtual system to assess the impact.

Infiltrating Insider
Rootkits are not like email-borne viruses, spreading indiscriminately. They're often employed by trusted insiders to steal information and spy on your staff. In these cases, you'll probably want to quietly monitor any suspects to gather evidence.

This calls for stealth, using host-based agent technology to monitor activity and collect evidence over time. This agent technology is usually installed like a service on a computer, but will take steps to hide itself from the user of the machine. It may have the ability to take screen shots, sniff keystrokes and monitor emails. Your response team can digitally wiretap computers, track a suspect and collect evidence. This may be the only option to detect collusion, reveal what data is being targeted, and uncover how deep the threat is. This can be very important to law enforcement agencies, your insurance company and your board of directors to ensure that you have a forensics trail for prosecution, if it comes to that. It will also show the proactive steps you've taken to ensure government compliance.


Putting it in Reverse
Reverse engineering dissects rootkits, but a lack of tools makes it a chore for most organizations.

Sometimes the best evidence can be obtained by reverse engineering. A rootkit or backdoor malware usually contains special commands. A remote attacker may send commands to have the malware copy critical files, email or even connect to a corporate database. These features can become apparent once the code is reverse engineered. Forensic analysts can determine what capabilities the malware has, and thus deduce some of the intentions of the attacker. This can help characterize the threat and give clues as to other systems that may also be infected. Reverse engineering can help you:
  1. Develop a single signature to detect all the variants of polymorphic malware.
  2. Learn how to decrypt captured packet streams if a rootkit encrypts its network communications. This can be used to reveal what has been exfiltrated, or to recover what kinds of remote control commands have been sent.
  3. Understand how the malware penetrated the network--which software exploits were used, what parts of the system are still vulnerable to attack, etc.
Yet, for all its power, reverse engineering is not an adopted practice in most organizations. The reason is that there aren't very many tools that make reverse engineering easier. The current state-of-the-art consists of esoteric and poorly documented tools that require users to read cryptic disassembled instruction code. At the recent CEIC security conference, only one vendor was offering a reverse engineering product for malware (see figure, below).

There is a perceived need for reverse engineering, but the idea is just starting to take hold. Fortunately, digital forensics is a rapidly growing market and many new products are being offered that feature remote forensics, "live" forensics, malware behavior analysis, automated reverse engineering, and covert/stealth monitoring of insider threats.

HBGary's Inspector reverse engineering product traces code while bypassing anti-debugging tricks to capture program instructions and live memory. Dynamic analysis provides information about malware behavior, and runtime dataflow analysis records how data propagates in memory.

Host-based wiretaps are especially effective at getting around encryption. Even drive-based forensics can be thwarted by encryption. But, regardless of the encryption used, the data must exist unencrypted at some point, usually in memory, where host-based "wiretaps" can sniff it before it's encrypted.

Be aware, however, that the attacker may detect if he or she is being monitored, whether you're dealing with a cagey admin or skilled remote hacker. Be prepared to use sophisticated stealth tools--few of the many commercial enterprise host-based monitoring tools available employ effective stealth capabilities. Many are child's play to bypass. They use schemes like renaming the process or service to something that doesn't look suspicious.

A Tough Row to Hoe
Rootkits are state-of-the-art backdoors, designed to defy detection and removal as they continue to spy and steal. Detection solutions are woefully behind, and evidence-collection tools are inadequate. There is a strong reliance on human methods to detect and mitigate the rootkit threat, and these human methods do not scale well.

A well-made rootkit will likely go undetected, so infections are found well after the fact. This means forensics analysis is very important in order to assess damages and mitigate the threat elsewhere in the network. The best defense is blended, using more than one detection solution and investing in a lab that can reverse engineer and collect behavior from captured malware. The ability to remotely assess systems, capture programs and image drives is critical for your scalability. Ultimately, rootkit detection remains a human problem, so the challenge is really about scaling and noise reduction to make your human teams more effective.


Dig Deeper on Malware, virus, Trojan and spyware protection and removal