Virtualization server security best practices

Virtualization changes the game for enterprise IT, but security doesn't have to be a barrier to implementation.

Five years from now, few enterprises will use "real" computers. Everything done with databases, Web applications or file shares will be intercepted and emulated by virtualization software, allowing one rack-mount server to act like 10.

Virtualization is inescapable; it's the most important new force in enterprise IT since IP networks.

A security professional could be forgiven for feeling a lingering sense of dread about the implications of this trend. On internal networks, virtualization is redrawing the map, taking servers and applications that were once separated by hardware and network filtering and cramming them onto the same blade server. No change that far-reaching could come without security challenges, and the products we're using to make those changes aren't even 10 years old.

The good news is that virtualization is a win for enterprise security. Patching, staging, deployment and change management--chronic headaches for IT security--get easier in virtualized data centers. The bad news is, before virtualization solves those problems for us, we've got challenges to overcome. In no particular order, here are five do's and don'ts for avoiding virtualization pitfalls.

DON'T let your enterprise silo virtualization security
An enterprise typically has Windows administrators managing Windows security, Unix administrators for Unix security and storage administrators to keep the SAN locked down. Thinking that organization is also going to work for VMware ESX clusters is a fatal mistake.

"The organizational impact of virtualization is profound," says Christofer Hoff, chief security architect at Unisys and an expert on virtualization security. "A lot of companies are getting caught flat-footed" by virtualization security, he says, and networking teams are throwing up deployment roadblocks. So long as it's only the VMware admins left holding the ball on security, nobody else has any skin in the game. What you're left with is a fragmented, half-deployed architecture where security is an afterthought.

Think back to the 1990s, when enterprise switching and VLANs emerged. Lack of cooperation and a poor game plan for security left us where we are today with open, uncontrolled networks where one hacked help desk computer can threaten mainframes and storage networks. Don't let that happen again.

Virtualization gives enterprises a second chance to get the IT security playbook right. Server admins should have a plan for staging, deploying and patching virtual machines. Network admins should have a plan for keeping access rules tight and consistent around physical servers and between guest operating systems. And security teams should have policies in place to audit configuration and deployment.

DO practice segmentation on your physical VM servers
Some guest hosts are going to handle sensitive data, such as credit cards or protected health information. Others won't. Don't let these two types of VMs share the same hardware.

For proof of why segmentation is critical, ask Tavis Ormandy. Last summer, Ormandy published an in-depth study of virtualization security with a report documenting "iofuzz," a tool that uncovered vulnerabilities in practically every virtual machine hypervisor he tested. When it comes to virtualization, "x86 is tough to get right," he says. For Ormandy, the idea that developers will have an easier time writing secure hypervisors than secure operating system kernels is a tough sell. Ouch; it took more than a decade to lock down Windows.

What does a hypervisor vulnerability mean? Just this: someone with access to any one of your VMs can "jailbreak" into the host and compromise the rest of them. Isn't that a good enough reason to keep sensitive VMs on separate hardware from testing VMs?

And jailbreaking vulnerabilities aren't the end of the problem. Consider all the network security mechanisms you've put in place to guard your data centers. Or, don't, because when it comes to traffic running on "virtual switches" between guest hosts on the same hardware, none of it matters. "Trying to replicate high-availability network security given today's virtual switching introduces really nasty performance and availability issues," says Hoff.

What should you do? The answer is technically simple but organizationally difficult. Enterprises need to figure out what their security domains are, gaining a sense of what their most sensitive data is. Then, machines that handle that data need to be kept on isolated hardware, whether it's efficient to do so or not.

DON'T ignore the risks of virtualization add-on services
Aside from the issues associated with network security and virtual machines, companies need to think through virtual machine migration. Migration features like VMware's "VMotion," which allows a VM to hop from one hardware platform to another without downtime, are one of the ideas that get enterprises interested in virtualization in the first place. But they can play havoc with security.

Many IT teams are relying on virtual machines as "virtual security appliances," through which all traffic in and out of application VMs must be routed. That can be a problem, explains Hoff. Those VMs "don't take well to being 'vmotioned,' because the things that protect them don't move with the VM."

It gets trickier. Last year, a research team at the Univer-sity of Michigan published a report at USENIX demonstrating attacks against migration in VMware and Xen, the two most popular platforms. Their modus operandi: rewriting virtual machines on the fly as they crossed the network. By the time they landed at their destination server, operating systems that had been secure just moments before were backdoored.

Don't overlook backup, either. Checkpointing and snapshoting capabilities (see "Virtualization Lexicon," below) in virtualization software are giving rise to a cottage industry of special-purpose products that promise to streamline backup storage for systems like VMware ESX. There is no more sensitive function in IT than backup and disaster recovery, which handle vast quantities of protected information. Be sure your backup vendor understands that.

How does a company know if its VMware backups are safe? It should ask its vendor if a third party has tested the security of its product, and if so, what did the tester find? Vendors that skip this step invite disaster. Network backup products want the keys to log in to all your virtualization servers; if they have bugs, attackers can steal those keys and with them every virtualized host in your enterprise. Caveat emptor.

DO consider virtualization security products carefully and critically
The virtualization security product market is an emerging industry to keep a watchful eye on. "It's a splashy area of security," says Marty Roesch, CTO of network security vendor Sourcefire and creator of the Snort intrusion detection project. "People are asking us what we can do." But he questions whether that's the right battle for enterprises to fight.

Roesch asks why the intra-VM traffic, running between guest operating systems on the same hardware, is "so much more important than the traffic at the switching and access layer."

For years, enterprises have struggled to get security policies right on the internal network. Perhaps zealous efforts to get security deployed in every virtual switch shouldn't take priority over solving security on real networks.

"I have to ask if it's better to deploy security in 200 server blades, or whether your threat is coming from outside in. How is it better to have sensors watching each blade individually, as opposed to watching one at the uplink?" Roesch says.

Because it's cheaper, counters Aaron Bawcom, vice president of engineering at Reflex Security, a virtual network security company. "We've seen customers with hundreds of locations, each with multiple point-of-sale systems rolled up into a few servers handling the IT infrastructure for that location. Have you looked at the cost of deploying just a firewall at a thousand sites?" The cost of deployments like these is so high, he says, companies are simply paying the fines from credit card industry audit violations rather than re-architecting the network.

Rather than deploying hardware to every branch, Bawcom wants enterprises to consider exploiting virtualization to consolidate servers at branch offices. Once you're managing just a single physical server and three virtualized guests, you can implement network security simply by adding it to the virtual switch. "With hardware appliances, there's a barrier of ROI that you can't get past. When you virtualize security, you can deploy more of it for less cost and more value," he says.

One area in which Roesch and Bawcom agree is security monitoring. "Network visibility thrives when you deploy it as virtual appliances at the hypervisor level," says Roesch, "because when you have tools to distill it for you, the more information you can get, the better." For Bawcom, virtualization also creates new opportunities for management, allowing enterprises to get a top-down map view of their systems and to go back in time to see what's changing.

None of these benefits are free, however. "Virtualization doesn't reduce security costs," argues Hoff. "You're still deploying the same agents everywhere; same intrusion prevention, same antivirus." The implication is that the flexibility of virtualized environments can also be their undoing. "Virtual security appliances are being asked to screen every single traffic flow. Just when you think you've got the memory and CPU constraints for that worked out, 10 more VMs get migrated to that server. It's very difficult to forecast how much throughput you're going to need," he says.

There's an elephant in this virtual security room. Far and away the most popular provider of enterprise virtualization is VMware, and VMware has not been standing still on security. VMware is in position to make virtualization security a feature instead of a product, but for now, the company is giving mixed signals. Its recently announced VMsafe initiative promises to make VMware hypervisors more accessible to third-party vendors. But last year, it purchased Determina, an up-and-coming host security company. VMware now finds itself supporting a top-caliber security research team staffed with researchers such as Alex Sotirov and Oded Horovitz, both famous vulnerability hunters, neither of whom is just sitting around doing nothing.

Ultimately, enterprises need to apply a healthy dose of skepticism when it comes to virtualization security products. Ormandy puts it simply: "People still believe that virtualization can be a security silver bullet, which does not reflect current reality." Virtualization security products can offer opportunities for more widespread network security coverage, but those opportunities should be clear, compelling and immediate before an enterprise acts on them.

DON'T let virtualized malware keep you up at night
Here's something that isn't clear, compelling and immediate: the threat of virtualized malware. What's virtualized malware? It is Trojan horse rootkit software that exploits hypervisor technology to hide itself "above" the infected operating system. The grim promise of virtualized malware is rootkits and botnets that are undetectable.

Anyone who follows security carefully has probably heard about virtualized rootkits. As a news story, it writes itself: virtualization is hot, and security attacks always make good reading. But how much of a problem are virtualized rootkits in the real world? Not much at all; they're essentially never seen in the wild.

So why aren't we seeing a new wave of malware taking advantage of virtualization capabilities? Researchers developing proof-of-concept rootkits might argue it's because we're not looking for them, or able to find them with our current tools. But that might not be the case.

Last year, this writer worked with Nate Lawson from Root Labs and Peter Ferrie from Symantec to develop techniques for detecting virtualized rootkits. We found so many ways to do it that we doubt the pursuit of "undetectable" virtualized rootkits is a good strategy. The team's key finding was that virtualization does a great job of hiding itself from applications that aren't looking for it, and that's enough to keep the lights on and the hard drives spinning. But when you look closely, illicit hypervisors leave telltale signs that are extraordinarily hard to conceal.

It's not all good news. The rootkit threat is very real; it's just more likely to bite at the application layer. There are only a few virtualization platforms for a rootkit to hide in; we can audit those. But there are tens of thousands of applications, each with hiding places for backdoors and rootkits. Without a doubt, enterprises will need to remain vigilant in the era of virtualization.

@exb

				implementation
				Insurance Policy Esurance makes security a priority as it steps into the world of virtualization. When Esurance embarked on virtualization, it undertook the project like any other: securely. "Security is part of our DNA at Esurance," says Marjorie Hutchings, director of Internet operations at the San Francisco-based online insurance company. "No matter what we implement, security is at the forefront of each project." Esurance, whose television ads feature Erin, a pink-haired cartoon crime fighter, deployed VMware in its pre-production environment and more recently virtualized its enterprise directory services. In implementing virtualization, the company adopted the same types of security measures it has in its physical infrastructure, Hutchings says. That includes antivirus software, strict administrative controls, and monitoring for any kind of configuration changes to guard against misconfigurations. Virtualization makes it easy to enforce a hardened server build, she says. The company also isolates the management network in the virtualized infrastructure, and keeps virtual machines with sensitive data separate from others. "We make sure to pay extreme attention to securing the virtual drive image as well as the virtual machine template," Hutchings adds. Virtualization allows the fast-growing insurance firm to bring up additional development environments quickly, easily and properly configured while saving on hardware. The technology also helps the company with its green initiative, Hutchings says: "Virtualization allows us to save power and energy and reduces our carbon footprint." Esurance, which has more than a half million policyholders in 28 states, hopes to expand its use of virtualization into its production environment, possibly next year. --Marcia Savage