News Stay informed about the latest enterprise technology news and product updates.

Researchers develop malware detection for hypervisor security

New software called HyperSentry stealthily detects malware in the hypervisor running virtual environments and alerts administrators to respond to a compromise.

Researchers at North Carolina State University and IBM are developing software to protect virtual environments by focusing on threat detection in the hypervisor, a feat that up until now has been nearly impossible.

Once an attacker compromises a hypervisor they have complete control of the system. They can alter the guest systems and steal sensitive data from them.

Peng Ning,
professor of computer scienceNorth Carolina State University

Called HyperSentry, the software measures the integrity of hypervisors in runtime. Peng Ning, professor of computer science in the College of Engineering at NC State and co-author of a paper describing the HyperSentry research, said the goal is to better protect virtual environments by focusing on detecting malware that can bypass traditional security technologies.

"Once an attacker compromises a hypervisor they have complete control of the system," Ning said. "They can alter the guest systems and steal sensitive data from them."

The hypervisor or virtual machine manager is the brains of a virtual machine and manages the sharing of hardware between multiple guest systems. Initially, the code-base of hypervisors had been small and seen as relatively secure, but the code-base has been increasing to support more systems and as a result there have been increased vulnerabilities, Ning said. Threats against the hypervisor have been theoretical. Some security researchers have demonstrated ways attackers can defeat the hypervisor, creating a backdoor to gain control of the guest machines.

"Although vulnerabilities have been quickly patched, we can imagine that new ones will be discovered," Ning said.

The software resides in the memory in the platform management interface of a server and uses the system management mode of the processor. An agent that remains undetectable is used to examine the hypervisor. It inspects the program memory and the registers inside the CPU for any anomalies that could be malware. If anything out of the ordinary is detected, the software sends an alert to an IT administrator.

"It looks at the code of the hypervisor to see if any part of the software has been changed," Ning said. "It also looks to see if the hypervisor has enforced isolation between different virtual machines as it should have."

The HyperSentry software runs on existing hardware and firmware and remains isolated from the hypervisor, Ning said. This keeps a compromised hypervisor from detecting the software's measuring process, he said.

"If the compromised hypervisor knows it is being looked at, an attacker can reinstall the hypervisor to the originally known good state so when antimalware software looks at the hypervisor it will see the same good software there," Ning said. "It won't be able to see the hypervisor has been compromised."

HyperSentry has been tested successfully on an open source Xen hypervisor. Ning said the software shows promise for use in real world systems, but it could take years before it is available in enterprise environments. Tests have shown it tamper-proof against any attacks that can compromise the hypervisor. Future research will focus on effectively measuring the integrity of other hypervisors and developing ways to make it easily deployable by IT teams by installing a driver on their firmware platform.

"Measuring the integrity to answer this with 100% confidence is still an open problem," Ning said. "We know how to measure the code part, the static data part and some of the special properties as well, but in general we don't know how to measure arbitrary integrity properties yet.""

Dig Deeper on Virtualization security issues and threats

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.