This article can also be found in the Premium Editorial Download "Information Security magazine: Five crucial virtualization do's and don'ts."
Download it now to read this article plus other related content.
Virtualization changes the game for enterprise IT, but security doesn't have to be a barrier to implementation.
Virtualization is inescapable; it's the most important new force in enterprise IT since IP networks.
A security professional could be forgiven for feeling a lingering sense of dread about the implications of this trend. On internal networks, virtualization is redrawing the map, taking servers and applications that were once separated by hardware and network filtering and cramming them onto the same blade server. No change that far-reaching could come without security challenges, and the products we're using to make those changes aren't even 10 years old.
The good news is that virtualization is a win for enterprise security. Patching, staging, deployment and change management--chronic headaches for IT security--get easier in virtualized data centers. The bad news is, before virtualization solves those problems for us, we've got challenges to overcome. In no particular order, here are five do's and don'ts for avoiding virtualization pitfalls.
DON'T let your enterprise silo virtualization security
"The organizational impact of virtualization is profound," says Christofer Hoff, chief security architect at Unisys and an expert on virtualization security. "A lot of companies are getting caught flat-footed" by virtualization security, he says, and networking teams are throwing up deployment roadblocks. So long as it's only the VMware admins left holding the ball on security, nobody else has any skin in the game. What you're left with is a fragmented, half-deployed architecture where security is an afterthought.
Think back to the 1990s, when enterprise switching and VLANs emerged. Lack of cooperation and a poor game plan for security left us where we are today with open, uncontrolled networks where one hacked help desk computer can threaten mainframes and storage networks. Don't let that happen again.
Virtualization gives enterprises a second chance to get the IT security playbook right. Server admins should have a plan for staging, deploying and patching virtual machines. Network admins should have a plan for keeping access rules tight and consistent around physical servers and between guest operating systems. And security teams should have policies in place to audit configuration and deployment.
DO practice segmentation on your physical VM servers
For proof of why segmentation is critical, ask Tavis Ormandy. Last summer, Ormandy published an in-depth study of virtualization security with a report documenting "iofuzz," a tool that uncovered vulnerabilities in practically every virtual machine hypervisor he tested. When it comes to virtualization, "x86 is tough to get right," he says. For Ormandy, the idea that developers will have an easier time writing secure hypervisors than secure operating system kernels is a tough sell. Ouch; it took more than a decade to lock down Windows.
What does a hypervisor vulnerability mean? Just this: someone with access to any one of your VMs can "jailbreak" into the host and compromise the rest of them. Isn't that a good enough reason to keep sensitive VMs on separate hardware from testing VMs?
And jailbreaking vulnerabilities aren't the end of the problem. Consider all the network security mechanisms you've put in place to guard your data centers. Or, don't, because when it comes to traffic running on "virtual switches" between guest hosts on the same hardware, none of it matters. "Trying to replicate high-availability network security given today's virtual switching introduces really nasty performance and availability issues," says Hoff.
What should you do? The answer is technically simple but organizationally difficult. Enterprises need to figure out what their security domains are, gaining a sense of what their most sensitive data is. Then, machines that handle that data need to be kept on isolated hardware, whether it's efficient to do so or not.
DON'T ignore the risks of virtualization add-on services
Many IT teams are relying on virtual machines as "virtual security appliances," through which all traffic in and out of application VMs must be routed. That can be a problem, explains Hoff. Those VMs "don't take well to being 'vmotioned,' because the things that protect them don't move with the VM."
It gets trickier. Last year, a research team at the Univer-sity of Michigan published a report at USENIX demonstrating attacks against migration in VMware and Xen, the two most popular platforms. Their modus operandi: rewriting virtual machines on the fly as they crossed the network. By the time they landed at their destination server, operating systems that had been secure just moments before were backdoored.
Don't overlook backup, either. Checkpointing and snapshoting capabilities (see "Virtualization Lexicon," below) in virtualization software are giving rise to a cottage industry of special-purpose products that promise to streamline backup storage for systems like VMware ESX. There is no more sensitive function in IT than backup and disaster recovery, which handle vast quantities of protected information. Be sure your backup vendor understands that.
How does a company know if its VMware backups are safe? It should ask its vendor if a third party has tested the security of its product, and if so, what did the tester find? Vendors that skip this step invite disaster. Network backup products want the keys to log in to all your virtualization servers; if they have bugs, attackers can steal those keys and with them every virtualized host in your enterprise. Caveat emptor.
DO consider virtualization security products carefully and critically
Roesch asks why the intra-VM traffic, running between guest operating systems on the same hardware, is "so much more important than the traffic at the switching and access layer."
For years, enterprises have struggled to get security policies right on the internal network. Perhaps zealous efforts to get security deployed in every virtual switch shouldn't take priority over solving security on real networks.
"I have to ask if it's better to deploy security in 200 server blades, or whether your threat is coming from outside in. How is it better to have sensors watching each blade individually, as opposed to watching one at the uplink?" Roesch says.
Because it's cheaper, counters Aaron Bawcom, vice president of engineering at Reflex Security, a virtual network security company. "We've seen customers with hundreds of locations, each with multiple point-of-sale systems rolled up into a few servers handling the IT infrastructure for that location. Have you looked at the cost of deploying just a firewall at a thousand sites?" The cost of deployments like these is so high, he says, companies are simply paying the fines from credit card industry audit violations rather than re-architecting the network.
Rather than deploying hardware to every branch, Bawcom wants enterprises to consider exploiting virtualization to consolidate servers at branch offices. Once you're managing just a single physical server and three virtualized guests, you can implement network security simply by adding it to the virtual switch. "With hardware appliances, there's a barrier of ROI that you can't get past. When you virtualize security, you can deploy more of it for less cost and more value," he says.
One area in which Roesch and Bawcom agree is security monitoring. "Network visibility thrives when you deploy it as virtual appliances at the hypervisor level," says Roesch, "because when you have tools to distill it for you, the more information you can get, the better." For Bawcom, virtualization also creates new opportunities for management, allowing enterprises to get a top-down map view of their systems and to go back in time to see what's changing.
None of these benefits are free, however. "Virtualization doesn't reduce security costs," argues Hoff. "You're still deploying the same agents everywhere; same intrusion prevention, same antivirus." The implication is that the flexibility of virtualized environments can also be their undoing. "Virtual security appliances are being asked to screen every single traffic flow. Just when you think you've got the memory and CPU constraints for that worked out, 10 more VMs get migrated to that server. It's very difficult to forecast how much throughput you're going to need," he says.
There's an elephant in this virtual security room. Far and away the most popular provider of enterprise virtualization is VMware, and VMware has not been standing still on security. VMware is in position to make virtualization security a feature instead of a product, but for now, the company is giving mixed signals. Its recently announced VMsafe initiative promises to make VMware hypervisors more accessible to third-party vendors. But last year, it purchased Determina, an up-and-coming host security company. VMware now finds itself supporting a top-caliber security research team staffed with researchers such as Alex Sotirov and Oded Horovitz, both famous vulnerability hunters, neither of whom is just sitting around doing nothing.
Ultimately, enterprises need to apply a healthy dose of skepticism when it comes to virtualization security products. Ormandy puts it simply: "People still believe that virtualization can be a security silver bullet, which does not reflect current reality." Virtualization security products can offer opportunities for more widespread network security coverage, but those opportunities should be clear, compelling and immediate before an enterprise acts on them.
DON'T let virtualized malware keep you up at night
Anyone who follows security carefully has probably heard about virtualized rootkits. As a news story, it writes itself: virtualization is hot, and security attacks always make good reading. But how much of a problem are virtualized rootkits in the real world? Not much at all; they're essentially never seen in the wild.
So why aren't we seeing a new wave of malware taking advantage of virtualization capabilities? Researchers developing proof-of-concept rootkits might argue it's because we're not looking for them, or able to find them with our current tools. But that might not be the case.
Last year, this writer worked with Nate Lawson from Root Labs and Peter Ferrie from Symantec to develop techniques for detecting virtualized rootkits. We found so many ways to do it that we doubt the pursuit of "undetectable" virtualized rootkits is a good strategy. The team's key finding was that virtualization does a great job of hiding itself from applications that aren't looking for it, and that's enough to keep the lights on and the hard drives spinning. But when you look closely, illicit hypervisors leave telltale signs that are extraordinarily hard to conceal.
It's not all good news. The rootkit threat is very real; it's just more likely to bite at the application layer. There are only a few virtualization platforms for a rootkit to hide in; we can audit those. But there are tens of thousands of applications, each with hiding places for backdoors and rootkits. Without a doubt, enterprises will need to remain vigilant in the era of virtualization.
This was first published in June 2008