This content is part of the Essential Guide: Tackle endpoint defense issues to obtain the best endpoint security
Get started Bring yourself up to speed with our introductory content.

Endpoint protection: How to select virtualization security tools

Most virtualization security tools still follow dedicated agent models, but some technologies are starting to offload resources to a dedicated VM and leverage hypervisor APIs.

As more network infrastructure becomes virtualized in both private and public cloud environments, how is traditional...

endpoint security technology evolving to adapt?

Several virtualization security tools exist for host-based security monitoring and protection in virtual environments. The first endpoint security approach is fairly traditional; security teams often use either a standalone antivirus or host-based intrusion detection system/intrusion prevention system (HIDS/HIPS) agent in the virtual machine (VM), or antivirus and host-based monitoring that's been adapted for virtual infrastructure with hypervisor APIs.

Specialized tools are not as common for virtual environments. But some endpoint security technologies, such as Bit9 + Carbon Black, Mandiant (FireEye) and Guidance Software's EnCase platform, offer whitelisting and file integrity monitoring agents or endpoint forensics agents. Other endpoint protection tools, such as Bromium and Invincea, leverage virtualization capabilities, although this type of software is often found on traditional endpoints.

In virtual environments, where pooled resources are the norm, any virtualization security tools that drain system resources on a per-VM basis should be regarded as a potential risk to the whole virtualization ecosystem. In fact, much of the antivirus industry still has to adapt to accommodate VMs and the performance ramifications of virtualization's shared resource compute model. Examples of virtualization-friendly antivirus include Kaspersky Security for Virtualization, Bitdefender Security for Virtualized Environments, and Symantec Endpoint Protection. These antivirus tools have been optimized for performance and scheduling, offering more lightweight deployment options than usually found on traditional endpoints.

New architectures are emerging that tie an HIDS/HIPS VM to the hypervisor kernel, passing all traffic and activity through the VM for "cleaning." VMware's vShield Endpoint, a commercial product, is primarily an integrated interface and architecture that allows antimalware products like Trend Micro's Deep Security, Sophos Antivirus for vShield, and Intel Security's MOVE Antivirus (McAfee Management for Optimized Virtual Environment) to operate efficiently within the hypervisor. The architecture is very innovative -- a single VM is designated as the "antivirus/HIDS VM," and a low-level bus in the hypervisor kernel sends all traffic and data to be evaluated within that VM only. This saves a significant amount of overhead, because none of the production VMs require a heavy agent.

Key criteria for evaluation

The most important criteria for teams evaluating host-based security for VMs are compatibility, performance specifications, and scalability for agents and tools in the virtual environment. Security and operations teams should also evaluate how the tools will be integrated (or if they can be). Be sure to investigate whether the host-based security tools are compatible with virtualization management consoles like vCenter (VMware), System Center Virtual Machine Manager (SCVMM for Hyper-V) or XenCenter for XenServer. This type of integration is not that common. Most HIDS/HIPS and antimalware agents have a separate console already, but any integration capabilities should be thoroughly evaluated, especially if there's an operational need for consolidated management. Simple architectural considerations also apply. For example, will putting the HIDS/HIPS management console in a VM on the same hypervisor platform be a better use of resources? This may be the case in a cloud environment, especially if it's hosted elsewhere.

As an alternative, tools like Bromium and Invincea are two host-based security technologies that use virtualization to defeat attacks. Bromium, founded by Xen creator Simon Crosby, is a hybrid Type-I hypervisor that uses the Intel VT-X chipset virtualization to create a thin hypervisor layer under the actual OS (Windows, for example). Any malware or attacks on the system are intercepted by a local policy engine that can use the hardware layer for enforcement, almost emulating the idea of security researcher Joanna Rutkowska's Blue Pill rootkit in some ways. Invincea, on the other hand, leverages application virtualization within the OS with a policy "wrapper" around certain high-risk applications like browsers and email clients.

What's ahead for virtualization in endpoint security?

The market for virtualization endpoint security is evolving rapidly. Most tools follow the traditional model, using a dedicated agent. Some technologies are starting to offload resources to a dedicated VM and leverage hypervisor APIs to manage detection and prevention tasks. Still more endpoint security software makes use of the virtualization capabilities themselves, preventing attacks from successfully interacting with the hardware, memory or OS.

Dave Shackleford is the owner and principal consultant of Voodoo Security LLC; lead faculty at IANS; and a SANS analyst, senior instructor and course author. He previously worked as CSO at Configuresoft; as CTO at the Center for Internet Security; and as a security architect, analyst and manager for several Fortune 500 companies. He currently serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance.

This was last published in June 2015

Dig Deeper on Virtualization security issues and threats