Test labs are the ideal place to check theory against reality.
|Infrastructure Via eBay|
It's nice to buy new, but nothing matches the thrill of getting a $50,000 VPN concentrator for $72.47 (shipping included) on eBay. eBay has a world of problems: You can't get quantities; prices are unpredictable; and quality is haphazard. But, if you're not in a hurry and are willing to wait for the right item, eBay can save you up to 90 percent on test lab equipment. Volumes have been written about how to get the best deals on eBay, but buying for a test lab is different. Avoid premium-priced brands and look for equivalent products. Cisco Systems gets a premium for its products because of its technical and maintenance support, the assurance of a top-tier vendor and its commitment to supporting even the oldest gear. But none of that is important in a test lab. If you need a LAN-switching infrastructure, buy Extreme Networks gear instead. A used Cisco 2948G costs between $750 and $1,000, but the Extreme Summit 48, with essentially identical capabilities, better performance and quieter fans, goes for half that price.
Of course, eBay isn't always the right answer. Sometimes you do want the same gear in your lab as you have in your production network. But, when you can save money, why not give eBay a try?
— JOEL SNYDER
Equipment and software testing is a fact of life for the network security manager. When a pile of security and performance patches comes in from the great unknown, the test lab is your sanity check. When your RFP for new equipment hits the street, the test lab is where you verify whether hardware meets the specs. When planning changes to your security infrastructure, the test lab is where you experiment on your theories.
Magazine and online product reviews, analyst reports and white papers can help with product assessments, but a well-equipped test lab will give you the peace of mind that a product is right for your enterprise.
So what makes a good test lab?
For most network managers, building a test lab isn't a foreign task: It's just a smaller version of their production network. But, setting up a reliable testing environment means acquiring infrastructure, adding test equipment and establishing repeatable procedures to keep you from reinventing the wheel. Proper security testing starts with solid network infrastructure, and, because testing is an ongoing process, it pays to have a variety of equipment on hand.
Fortunately, network infrastructure is cheap — at least for testing purposes. With the exception of testing performance, you can live with off-the-shelf 10/100 switching equipment and slightly antiquated servers. If you can't scavenge from within the company, high-quality gear for pennies can be found at online auctioneer eBay, the de facto marketplace for used technology (see "Infrastructure via eBay," p. 40). Next, you need samples of all your production security infrastructure devices, especially firewalls. But, before charging up the company credit card, check with the manufacturer — some offer indefinite loans of demo units or significant discounts on cold-spare hardware. For software, where the vendor's cost is zero, negotiating a test lab license should be part of your standard purchasing procedure.
Finally, take a peek in your storerooms. You probably have a pile of dual-650 MHz rackmounted servers sitting around; they are perfect for testing. Your criteria here should be memory and ease of reconfiguration. In our lab, we use 2U Hewlett-Packard LPr servers. Dell, IBM and Compaq boxes are also excellent choices. As you stack up servers, you'll need a KVM (keyboard-video-mouse) switch. Forget the little eight-port ones you see around — they won't do. The Avocent Autoboot XP4040 is a recently retired — but still widely available — workhorse with virtually unlimited expansion capabilities. If you have a large corporate KVM switch, you might be able to use that. It's good practice to use these switches' built-in security features to keep the lab and production systems from being shown on the same console. This will also eliminate the potential of accidentally rebooting or, worse, reformatting the wrong system.
Systems generally fall into two categories: permanent and temporary. Perma-nent servers form the backbone of your lab's software services. For example, every lab needs a certificate authority to test PKI features. It pays to set this up once — not every time you need a digital certificate. You'll also find that authentication servers, such as RADIUS and LDAP, are repeatedly used and should be permanent fixtures. Any common corporate applications that can be easily replicated, such as Microsoft Exchange or Lotus Notes, along with your corporate Web server and database, should be running permanently in your test lab. You'll also need a file server to store software kits and other data.
|For Tests, Go Virtual|
Your permanent test lab should run on bare hardware; for everything else there's EMC's VMware.
This gem lets you run multiple virtual machines on a single server. It's also a security tester's best friend since it lets you quickly deploy a "clean" system for testing purposes. Having VMware images of multiple Windows, Linux and FreeBSD versions allows you, for example, to deploy a clean Windows Server 2003 — patched and preloaded with some common utilities — in just a few minutes.
VMware has become the de facto standard for QA testers around the world, and anyone seriously taking on the problem of patch management has already built a few servers to help in that task.
Although EMC would prefer that you buy the VMware GSX Server, you'll find that the additional capabilities of GSX Server (such as multiple remote managers) are only necessary on one or two of the largest and most heavily used servers. You can often manage the rest using the more affordable VMware workstation product.
Following its acquisition of Connec-tix, Microsoft released its own virtualization products — Virtual PC and Virtual Server 2005. As with VMware, the main difference between the two is management capabilities. In a production environment, you'll definitely want to go with Virtual Server 2005, which offers both stronger management and a data center focus. However, for the test lab, Virtual PC offers the core capabilities you need at a lower cost.
Many testers have a warm place in their hearts for Symantec's Ghost, which allows them to quickly build and deploy servers with virtually any OS on Intel hardware. Although Symantec has taken Ghost far into the world of system management and deployment for production systems, many of the bells and whistles added in recent versions don't add much for test lab functionality.
— JOEL SNYDER
Assemble a Toolkit
A test lab is invaluable as a sandbox for testing new configurations, interoperability and patches, and answering the two main questions plaguing every security manager: Does it work, and is it fast enough?
The open-source world has a wealth of outstanding testing tools. Your starting point should include applications such as Nessus (vulnerability scanner), Nmap (port scanner and system profiler), Netcat (TCP and UDP pipe tool), ping and traceroute. These products are well-documented and very reliable.
Beyond these basics, you'll want to branch out into more specific tools. In each category, you can find open-source tools of varying quality and usefulness to test coverage and performance of security products. The important thing to remember when using open-source tools is that they are often built to examine specific items. For example, several packages, such as "stick," focus on IDS evasion--a widely discussed issue largely irrelevant to most enterprise IDS deployments.
Security testing tools are designed to run on a mix of Windows and Unix systems. Laptops are great for basic functionality testing, especially Apple's OS X models; they give you easy access to standard tools such as Microsoft Office and Adobe Acrobat, yet have the Unix underpinnings required by most open-source tools. However, you'll find that having a few Windows-based laptops is critical, as many security products have Windows-only GUIs or require Internet Explorer.
At the server level, don't underestimate the number of systems you'll need. While some sort of virtualization (EMC's VMware or Microsoft's Virtual PC) should be one of your very first purchases, there's no substitute for a variety of different systems when it comes to running tests (see "For Tests, Go Virtual," p. 41).
Once you have your coverage and functionality test equipment set up, you'll want to dive into the world of performance testing, driving the hardware and software until it collapses under load. Your choice of equipment depends on what you need to test, but most test labs will want to start with Layer 2 and Layer 3 packet generation tools and Layer 7 application stress tools.
Layer 2 and Layer 3 packet generators--such as Spirent's SmartBits--create flows at virtually any speed. They'll generate the background noise that puts stress on the equipment you're testing so your real test tools can do their work. For example, sending a well-crafted attack through an IPS is interesting at one attack per minute, but it's a lot more interesting if the IPS sees the attack when it's simultaneously being pummeled by 100 Mbps of non-attack traffic. Layer 3 packet generators are very useful in testing VPN equipment by putting stress on the crypto engine and verifying the advertised capacity.
Layer 7 test tools, such as Spirent's Avalanche and Reflector, generate real application-layer streams over TCP and UDP. These are important because you'll often test equipment that looks at and modifies the data as it flows through the network.
Make It Repeatable
The most difficult--and important--part of testing is the development of a standard test plan. David Newman of Network Test, a benchmarking and network design consultancy, has three rules that form the game plan for anyone testing systems: Tests must be meaningful, stressful and repeatable.
Focus on what's meaningful--you need to inventory and survey your network to find out what is really going on. If you're testing firewalls, you need to know what protocols are flowing through your network, how many connections per second occur, how many octets flow over each connection, how many connections are open at once, and how complex your rule set is.
Settings such as logging can also have a huge effect on performance. Vendor tests are always done with logging turned off, but you'll want to test with logging turned on if your policy is to log everything to a SIM. As you gather this kind of data, keep a log of the different information.
Testing must stress the systems to their real configuration and performance limits. Knowing that a firewall can cope with existing load isn't interesting, but knowing how much your load can grow before the firewall collapses is invaluable. Stress isn't just a performance issue. If you have a 500-rule firewall, see what happens when you put 5,000 rules in it.
Can such a configuration be supported? If you're testing a high-availability system, how does it handle different failure scenarios? Don't just look at the ones that seem likely--like sudden power cord attenuation--but more stressful failures, like a network cable disconnecting and reconnecting several times a second for a few hours.
Determine when a system has failed. Most security devices have gradual system degradation rather than a hockey-stick plunge in the performance curve. As part of your test plan, set metrics ahead of time that define when a system is no longer acceptable. For example, a firewall that adds 2 milliseconds of latency is acceptable, 20 milliseconds is marginal and 200 milliseconds is unacceptable.
Finally, your tests have to be repeatable. You'll need to repeat the results in your own lab as vendors change product versions and as you make head-to-head comparisons. And, of course, you have to run every test several times to be sure that your results are consistent.
Reproducible tests require a solid, documented methodology. When you're preparing to do a test, capture the configuration of all the devices involved as well as the applicable test scripts. By being methodical, you'll not only be able to efficiently repeat tests, but you'll also be able to adapt them to new equipment and test scenarios.
Done right, testing gives you accurate and defensible answers about your product purchases and deployments. You'll know ahead of time that a product is right for your network and that a patch or upgrade won't melt down your systems or compromise your security infrastructure. For a minimal cost of time and money, testing lets you check your deployment strategy ahead of time and minimize the risks associated with change.