The first 'rootkits'
Such powerful tools, if modified and used maliciously, can help attackers effectively take control of an operating system while avoiding detection by regular antivirus programs. A collection of such tools, or a rootkit, first appeared at the beginning of the 1990s. Rootkits didn't really hit the headlines, though, until 2005, when Sony BMG Music Entertainment was caught including rootkit technology on various music CDs to prevent them from being copied. Although trying to protect its legitimate copyright of the CD's contents, this software actually altered the way in which the Microsoft Windows operating system worked. It also made it easy for malware authors to hide their files on compromised machines by simply using the Sony rootkit's ability to hide files, registry keys and processes.
Until recently there were five main types of rootkit: application, library, firmware, kernel and virtualized. Application-level rootkits replace regular application files or modify the behavior of an application by using hooks, patches or injected code. A similar technique is used by library rootkits, which commonly replace system files with versions that hide information about the attacker. As most antivirus scanners don't inspect firmware for code integrity, firmware rootkits can hide themselves in device or platform firmware with little chance of being detected. Kernel-level rootkits are among the most feared, though, as they operate at the same level as the operating system. Such access enables them to modify or subvert any requests made by software running on the system. Kernel-mode rootkits add or replace kernel code to hide their presence often via device drivers or loadable modules, as most operating systems don't enforce any security distinctions between the kernel and device drivers.
In recent years, researchers have been looking at ways to run rootkits outside of the operating system. Enter virtualized rootkits. Their modus operandi is very different from other rootkits as they modify a machine's boot sequence to load themselves instead of the original operating system. Once loaded into memory, this type of rootkit can load the original operating system as a virtual machine and intercept all hardware calls made by the "guest" OS, making the malware much more powerful and harder to detect. Two years ago researcher Joanna Rutkowska demonstrated a rootkit called Blue Pill, which used AMD's chip-level virtualization technology to hide itself.
Following the 2008 Black Hat Briefings security conference, a new type of rootkit emerged: the system management mode (SMM) rootkit. Developed by security researchers Shawn Embleton and Sherri Sparks of Clear Hat Consulting Inc., this rootkit hides itself by running in a protected part of a computer's memory that can be locked and rendered invisible to the operating system. This placement of the rootkit gives attackers a picture of what's happening in a computer's memory. Like many existing rootkits, it comes with keylogging and communications software, making the list of potential threats endless. One disadvantage of the SMM and virtualization rootkits from the hacker's perspective, however, is that they have to write the code expressly for the system they are attacking, meaning a rootkit designed to exploit one system hardware configuration may not necessarily work on a system configured differently. This may slow the proliferation of this type of rootkit.
So what can be done if you suspect that a rootkit is running on your machine? The fundamental problem with rootkit detection is that an infected operating system cannot be trusted. So, for example, when an antivirus program requests a list of all running processes or files in a directory from the OS, the results will not be accurate. This is why rootkit techniques are becoming increasingly popular with virus writers, as they can use similar cloaking techniques to hide their malware programs.
One way to locate a rootkit is to shut down the suspect computer and then check it by using a second "trusted" system, mounting the hard drive of the infected system as a resource. The hard drive can then be gainfully scanned, as a non-running rootkit cannot actively hide its presence. Many system administrators actually prefer to simply save data files, reformat the hard drive and use imaging software to install a clean OS on the infected machine instead of spending time and effort to try to locate and remove the rootkit. If a reinstall is not a viable option, boot the computer with a clean copy of the operating system using tools such as BartPE and Windows Preinstallation Environment (PE). This enables examination and replacement of the affected system files while keeping the underlying systems intact.
There are several programs available that try to detect rootkits, including Microsoft's RootkitRevealer. The program bypasses the operating system and analyzes the underlying structures in the file system, comparing them against expected values. If you've had the presence of mind to fingerprint your OS -- that is, calculate checksums to uniquely identify every file -- any critical files altered by a rootkit can be found by comparing message digest values. This technique can be used to detect firmware rootkits, too. You will, of course, have to re-fingerprint all subsequent changes made to your system.
The IT industry is still searching for the best set of methods to combat the threat posed by these new "undetectable" rootkits. As with computer viruses, the arms race between malware and detection code-writers will be an ongoing struggle. On the bright side, while rootkits have evolved tremendously since their modest beginnings, any of today's new malicious rootkits are ultimately harmless if never given the opportunity to infect a target system.
Rootkits are like many other IT security threats in that prevention is better than cure, so ensure enterprise systems have up-to-date patches and are protected from likely attack vectors. Install software only from trusted sources, and make sure security polices on acceptable usage are enforced, particularly at the end-user level. This last point will be critical when it comes to preventing SMM rootkits taking hold as many of the tools I've mentioned will be blind to them.
About the author:
Michael Cobb, CISSP-ISSAP is the founder and managing director of Cobweb Applications Ltd., a consultancy that offers IT training and support in data security and analysis. He co-authored the book IIS Security and has written numerous technical articles for leading IT publications. Mike is the guest instructor for several SearchSecurity.com Security Schools and, as a SearchSecurity.com site expert, answers user questions on application security and platform security.