Black Hat 2007: Rootkit hunters caught in cat-and-mouse game

Article

Black Hat 2007: Rootkit hunters caught in cat-and-mouse game

Dennis Fisher, Executive Editor
LAS VEGAS -- No malware, let alone a virtualized rootkit, is undetectable.

That was the message delivered loud and clear Wednesday at the Black Hat USA Briefings. A team of well-known security researchers led a session on the methods they believe would be effective in finding virtualized rootkits, such as

    Requires Free Membership to View

    Login

    SearchSecurity.com members gain immediate and unlimited access to breaking industry news, virus alerts, new hacker threats, highly focused security newsletters, and more -- all at no cost. Join me on SearchSecurity.com today!

    Michael S. Mimoso, Editorial Director

    By submitting your registration information to SearchSecurity.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchSecurity.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

Joanna Rutkowska's infamous Blue Pill or Dino Dai Zovi's Vitriol. The researchers outlined a number of techniques for detecting traces of such a rootkit's activity, including side-channel attacks, finding hypervisor bugs and looking for errors caused by the malware.

"You're basically stuck in a cat-and-mouse game in which the attacker designs some code, you look for characteristics of that code and detect it, and then it all repeats in a big cycle," said Nate Lawson, principal at Oakland, Calif.-based Root Labs, and one of the co-presenters of the session. "We've seen this before and people have always found countermeasures, and we expect that will continue the same way."

From left: Peter Ferrie, Symantec; Nate Lawson, Root Labs; Tom Ptacek, Matasano Security. Click to enlarge.
The presenters, who also included Thomas Ptacek of New York-based Matasano Security, Dai Zovi and Peter Ferrie of Cupertino, Calif.-based Symantec Corp., focused much of the talk on the properties of Blue Pill and the ways in which they would expect it to behave on a compromised system. Rutkowska, a well-known researcher based in Poland, gave a talk on the hypervisor rootkit at Black Hat in 2006, causing quite a stir. But she has not talked much about the exact features and functions of Blue Pill since then, and her claims of it being completely undetectable have drawn a lot of criticism from other researchers.

"We're really interested in debunking that claim," Ptacek said.

The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable.
Nate Lawson
Root Labs
The crux of the presenters' criticism of Blue Pill is that it attempts to emulate the entire architecture of an x86 machine, instead of just certain portions of the operating system as a conventional kernel-mode rootkit would. That ambitious design is exactly what makes Blue Pill detectable, Lawson said. Because it has to emulate so many difference components, it is bound to leave traces somewhere.

One of the methods Lawson outlined for detecting a virtualized rootkit involves observing changes in the Translation Lookaside Buffer (TLB), a cache in the CPU. When something causes a virtual machine to exit, the hypervisor leaves traces of its presence in the TLB. So, Lawson said, one way to detect a hypervisor rootkit would be to cause it to somehow exit, and then read the TLB and look for changes.

But, Lawson and Ptacek conceded, there's nothing stopping the malware author from writing a feature to detect the "rootkit detector," which Lawson said leads back to the familiar attacker-defender cat-and-mouse game.

"What you end up with is the same cycle that we see with AV engines and viruses, where I look at the latest version of your code, find ways to detect it and then you write a new version and we start all over again," Lawson said. "The reality is, there's no absolute endgame here. The malware authors can't make something that's 100% undetectable and I can't write a detector that makes all malware detectable."

Special Black Hat coverage

Check out more of SearchSecurity.com's special news coverage of Black Hat USA 2007.
The team of researchers also discussed a few details of their own detection software, called Samsara, which they plan to release in the next few weeks. They will make the code for the tool freely available, and also intend to make a prototype hardware-based rootkit available for testing purposes.

The rootkit session originally was intended to be a live demo in which Rutkowska would load Blue Pill onto one of several clean Vista machines, and Ptacek and his co-presenters would load Samsara onto all of the PCs and try to detect the rootkit. But Rutkowska declined the offer and instead sat in the audience.