- Dave Shackleford, Voodoo Security
At many companies, it's time for a reality check: Security within their virtualized environments is not up to par with that of traditional physical networks and systems.
Why have enterprise security teams been slow to adopt the policy and process changes needed to securely manage virtualization platforms? Sometimes, it's the result of immature technology such as endpoint security that doesn't play nicely in the virtual data center. In other cases, security teams may not know "what they don't know," and a lack of maturity in implementing processes and procedures could be at fault.
As more enterprises virtualize their data centers, the need to implement fundamental security practices and technologies within these environments has increased. Many technologies are available to help IT security implement controls within virtual infrastructure, but security teams only partially use these tools. As computing workloads become more condensed, security teams should seriously evaluate virtualization-specific technology and integrate it more deeply into their data centers.
Poorly managed environments
The good news is that virtualization can streamline patching and configuration management, but strict inventory management is paramount to avoid virtual machine sprawl.
Within virtualized environments, numerous virtual machines are housed on a single physical system, a condition known as multi-tenancy. The hypervisor software is responsible for maintaining segmentation and isolation among virtual machines. This can be augmented with open source or commercial virtual network and virtual security appliances or add-ons.
Configuration management can actually be easier in virtualized environments. Using templates simplifies the deployment of new virtual machines. Template maintenance helps centralize configuration management tools and practices into one place. Microsoft, Citrix and VMware all offer tools and other options for creating and managing virtual machine lifecycles and configurations. While patching remains a challenge for many organizations, testing patches is often easier with virtual machines because you can take a snapshot of an image before testing and roll back when the testing is complete.
A major security issue in many environments, however, is a lack of sound inventory management related to virtual machines. The ease with which administrators and developers can create virtual machines leads to systems being provisioned with little thought to lifecycle management. Virtual machines are paused or turned off, and remain dormant for a period of time; however, the files associated with these machines still contain sensitive data. When and if the virtual machines become active again, they may be missing patches and critical configuration controls.
To properly manage the dynamic nature of virtual environments, operations teams need to update and align their change management processes to accommodate the rapid pace of change, while taking into account the dependencies of all the components in the virtualization stack -- storage, network and hypervisors.
Role and privilege management
Separation of duties also tends to suffer in virtual environments. Some IT organizations start off their deployments by letting an existing team of administrators and engineers design and implement the virtualization technology. This same team often manages all the various components of the virtual infrastructure over time. This lack of separation of duties can lead to accidental errors or poorly managed environments (not to mention, major issues if one of the virtualization team members goes rogue and becomes a malicious insider).
It's not uncommon for VMware administrators to perform all management activities using the default "Administrator" role. This same role is often given to other IT administrators who are managing storage and networking objects in the virtual environment. These administrators often operate with entirely too many privileges.
This trend is a significant step backwards in terms of network security. Some organizations are deploying central access gateways to their virtualization management tools, which include more robust role and privilege management. One such product is the Hytrust appliance that integrates with VMware's vCenter management platform.
Network architecture consolidation in virtual data centers can also lead to compromises that affect security. Because all virtual machines share hardware, the number of network interfaces in hypervisor servers or blades may dictate how converged the network traffic is, as well.
Most virtual environments carry normal production traffic, as well as management traffic to access and operate the virtualization components' storage traffic like iSCSI or fibre channel. In addition, they carry specialized operations traffic like dynamic virtual machine migration (called vMotion in VMware environments).
VMotion traffic contains the content of virtual machine (VM) memory sent in cleartext, which may include sensitive data and authentication credentials. Management traffic can include configuration details or other useful information to an attacker. Mingling this information with production traffic may increase the exposure of data within the environment if an attacker manages to gain access to it. Few organizations today are keeping all of these traffic types completely distinct in the virtual data center.
Better technical controls
Multiple changes have occurred in the realm of virtualization-specific technical controls over the last several years. Fortunately, some of the tools available to enterprise security teams have improved significantly, allowing IT to achieve comparable security levels within virtual environments.
All major firewall vendors now have virtualized options that are available as virtual appliances (specialized VMs). Check Point Software Technologies, Cisco, Fortinet, Juniper Networks and Palo Alto Systems, (to name several well-known vendors) have various models and form factors of virtual appliances that can integrate into the virtual infrastructure. VMware also has its own vShield line of virtual firewalls for both internal and perimeter monitoring, and traffic control.
Many security teams are not using virtual firewalls extensively, however. The functionality is there, but network and security teams still feel that their physical firewalls are capable of handling traffic control within the network. Management and governance of these appliances is another challenge. Some virtualization teams feel that they should manage all components within the virtual environment, even those related to networking or security.
Reality check: Virtual network security tools
When evaluating virtual appliances, enterprise teams should consider the following checklist to improve virtual network security:
• Determine whether you need a virtual firewall based on the need for traffic control to, from and within the virtualization or cloud infrastructure. In some cases, this may be required for compliance, or you may need more granularity in traffic control than existing physical firewall segments and architecture can provide.
• Make sure your virtual IDS/IPS or firewall vendor supports your major virtualization technologies. Currently, most vendors support VMware, with limited support for Xen and Hyper-V platforms.
• Be considerate of how much resource overhead the virtual appliances will take up -- they require memory, disk space and CPU.
• Consider issues of governance and management related to firewall and IDS/IPS management. In most cases, letting teams that currently manage firewalls and IDS/IPS manage the virtual models makes sense.
From a maturity standpoint, network IDS/IPS is comparable to virtual firewalls. This functionality is sometimes offered in the same virtual appliance (taking more of a unified threat management, or UTM, approach to network security). See "Reality Check: Virtual Network Security Tools," page xx.
Aside from standalone platforms, intrusion detection can be accomplished much more readily in virtualization environments because of the advanced features available in virtual switches. Many switches -- including those from Microsoft, VMware, Cisco and the Open vSwitch project -- allow NetFlow export, as well as port mirroring options to copy traffic to a dedicated intrusion detection appliance or external sensor.
Changes in endpoint security
One area that is changing dramatically is endpoint security. In most virtualized environments, traditional antivirus agents are too resource intensive, and many of the traditional endpoint products are not wholly optimized for use in virtual systems. New technologies from Intel Security (MOVE) and Trend Micro (Deep Security) offload endpoint security processing to a dedicated virtual machine, while leveraging native API calls and kernel integration to keep the processing requirements on VMs low.
Many enterprises have not shifted to these specialized endpoint security products, although most have started testing these tools or rolling out limited implementations. Key things to keep in mind for security teams looking at endpoint security options for virtual environments include the following:
- Check the level of integration the vendor's product has with your virtualization technology. Currently, VMware has the most support from newer antimalware and other endpoint security vendors. Microsoft and Citrix have less integration with third-party vendors.
- Be aware that newer "offloading" models have limitations and drawbacks. While performance may improve, the ability to perform real-time scanning and in-memory analysis for behavioral heuristics may suffer. Also, these products may not scale well in large environments.
- Ask vendors for customer references, as well as performance statistics and metrics related to operation within your endpoint types and virtualization technology.
Hypervisor log collection
Logging and monitoring within virtual environments is currently very immature in many organizations, as well. The biggest issue is log collection from hypervisor platforms -- many simply aren't doing it or don't do enough of it. Hypervisor platforms produce a wide variety of logs that provide information on who is using the platform, what kinds of actions are taking place, performance and behavior statistics, and more. For example, VMware's ESXi hypervisor includes the following logs that security and operations teams should be centrally collecting:
- /var/log/syslog.log - generic Syslog file
- /var/log/auth.log - authentication and security events
- /var/log/vmkernel.log - miscellaneous hypervisor information
- /var/log/hostd.log - Master ESXi service log
- /var/log/vpxa.log - vCenter management agent logs
Most Linux-based hypervisors (Xen, KVM, VMware) natively include some sort of Syslog daemon, which makes log collection and aggregation simpler. In Hyper-V environments, logging agents for Windows Event Viewer logs should be used, although Microsoft System Center tools can retrieve log and event data from distributed hypervisors, too. Security teams should gather all relevant logs from all hypervisors in their environments and integrate this information into central log management and SIEM platforms for analysis and correlation.
Finally, the gap between the security technologies implemented within the physical and virtual environments still needs to be addressed at most companies. Many IT and security teams have been slow to adopt the numerous policy and process changes that are needed to better manage virtualization platforms and virtual machines. As investment in virtual data centers increases, enterprise security teams should seriously evaluate virtualization-specific security tools, and CISOs should implement and improve policies and procedures related to virtualization management and operations. Security within virtual environments is not up to par with that of physical systems and networks, but as the technology evolves and policies are put in place, that doesn't have to be the case.
About the author:
Dave Shackleford is the owner and principal consultant of Voodoo Security LLC; lead faculty at IANS; and a SANS analyst, senior instructor and course author. He previously worked as CSO at Configuresoft; as CTO at the Center for Internet Security; and as a security architect, analyst and manager for several Fortune 500 companies. He currently serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance.