Equipment and software testing is a fact of life for the network security manager. When a pile of security and performance patches comes in from the great unknown, the test lab is your sanity check. When your RFP for new equipment hits the street, the test lab is where you verify whether hardware meets the specs. When planning changes to your security infrastructure, the test lab is where you experiment on your theories.
Magazine and online product reviews, analyst reports and white papers can help with product assessments, but a well-equipped test lab will give you the peace of mind that a product is right for your enterprise.
Fortunately, network infrastructure is cheap — at least for testing purposes. With the exception of testing performance, you can live with off-the-shelf 10/100 switching equipment and slightly antiquated servers. If you can't scavenge from within the company, high-quality gear for pennies can be found at online auctioneer eBay, the de facto marketplace for used technology (see "Infrastructure via eBay," p. 40). Next, you need samples of all your production security infrastructure devices, especially firewalls. But, before charging up the company credit card, check with the manufacturer — some offer indefinite loans of demo units or significant discounts on cold-spare hardware. For software, where the vendor's cost is zero, negotiating a test lab license should be part of your standard purchasing procedure.
Finally, take a peek in your storerooms. You probably have a pile of dual-650 MHz rackmounted servers sitting around; they are perfect for testing. Your criteria here should be memory and ease of reconfiguration. In our lab, we use 2U Hewlett-Packard LPr servers. Dell, IBM and Compaq boxes are also excellent choices. As you stack up servers, you'll need a KVM (keyboard-video-mouse) switch. Forget the little eight-port ones you see around — they won't do. The Avocent Autoboot XP4040 is a recently retired — but still widely available — workhorse with virtually unlimited expansion capabilities. If you have a large corporate KVM switch, you might be able to use that. It's good practice to use these switches' built-in security features to keep the lab and production systems from being shown on the same console. This will also eliminate the potential of accidentally rebooting or, worse, reformatting the wrong system.
Systems generally fall into two categories: permanent and temporary. Perma-nent servers form the backbone of your lab's software services. For example, every lab needs a certificate authority to test PKI features. It pays to set this up once — not every time you need a digital certificate. You'll also find that authentication servers, such as RADIUS and LDAP, are repeatedly used and should be permanent fixtures. Any common corporate applications that can be easily replicated, such as Microsoft Exchange or Lotus Notes, along with your corporate Web server and database, should be running permanently in your test lab. You'll also need a file server to store software kits and other data.
Assemble a Toolkit
A test lab is invaluable as a sandbox for testing new configurations, interoperability and patches, and answering the two main questions plaguing every security manager: Does it work, and is it fast enough?
The open-source world has a wealth of outstanding testing tools. Your starting point should include applications such as Nessus (vulnerability scanner), Nmap (port scanner and system profiler), Netcat (TCP and UDP pipe tool), ping and traceroute. These products are well-documented and very reliable.
Beyond these basics, you'll want to branch out into more specific tools. In each category, you can find open-source tools of varying quality and usefulness to test
coverage and performance of security products. The important thing to remember when using open-source tools is that they are often built to examine specific items. For example, several packages, such as "stick," focus on IDS evasion--a widely discussed issue largely irrelevant to most enterprise IDS deployments.
Security testing tools are designed to run on a mix of Windows and Unix systems. Laptops are great for basic functionality testing, especially Apple's OS X models; they give you easy access to standard tools such as Microsoft Office and Adobe Acrobat, yet have the Unix underpinnings required by most open-source tools. However, you'll find that having a few Windows-based laptops is critical, as many security products have Windows-only GUIs or require Internet Explorer.
At the server level, don't underestimate the number of systems you'll need. While some sort of virtualization (EMC's VMware or Microsoft's Virtual PC) should be one of your very first purchases, there's no substitute for a variety of different systems when it comes to running tests (see "For Tests, Go Virtual," p. 41).
Once you have your coverage and functionality test equipment set up, you'll want to dive into the world of performance testing, driving the hardware and software until it collapses under load. Your choice of equipment depends on what you need to test, but most test labs will want to start with Layer 2 and Layer 3 packet generation tools and Layer 7 application stress tools.
Layer 2 and Layer 3 packet generators--such as Spirent's SmartBits--create flows at virtually any speed. They'll generate the background noise that puts stress on the equipment you're testing so your real test tools can do their work. For example, sending a well-crafted attack through an IPS is interesting at one attack per minute, but it's a lot more interesting if the IPS sees the attack when it's simultaneously being pummeled by 100 Mbps of non-attack traffic. Layer 3 packet generators are very useful in testing VPN equipment by putting stress on the crypto engine and verifying the advertised capacity.
Layer 7 test tools, such as Spirent's Avalanche and Reflector, generate real application-layer streams over TCP and UDP. These are important because you'll often test equipment that looks at and modifies the data as it flows through the network.
Make It Repeatable
The most difficult--and important--part of testing is the development of a standard test plan. David Newman of Network Test, a benchmarking and network design consultancy, has three rules that form the game plan for anyone testing systems: Tests must be meaningful, stressful and repeatable.
Focus on what's meaningful--you need to inventory and survey your network to find out what is really going on. If you're testing firewalls, you need to know what protocols are flowing through your network, how many connections per second occur, how many octets flow over each connection, how many connections are open at once, and how complex your rule set is.
Settings such as logging can also have a huge effect on performance. Vendor tests are always done with logging turned off, but you'll want to test with logging turned on if your policy is to log everything to a SIM. As you gather this kind of data, keep a log of the different information.
Testing must stress the systems to their real configuration and performance limits. Knowing that a firewall can cope with existing load isn't interesting, but knowing how much your load can grow before the firewall collapses is invaluable. Stress isn't just a performance issue. If you have a 500-rule firewall, see what happens when you put 5,000 rules in it.
Can such a configuration be supported? If you're testing a high-availability system, how does it handle different failure scenarios? Don't just look at the ones that seem likely--like sudden power cord attenuation--but more stressful failures, like a network cable disconnecting and reconnecting several times a second for a few hours.
Determine when a system has failed. Most security devices have gradual system degradation rather than a hockey-stick plunge in the performance curve. As part of your test plan, set metrics ahead of time that define when a system is no longer acceptable. For example, a firewall that adds 2 milliseconds of latency is acceptable, 20 milliseconds is marginal and 200 milliseconds is unacceptable.
Finally, your tests have to be repeatable. You'll need to repeat the results in your own lab as vendors change product versions and as you make head-to-head comparisons. And, of course, you have to run every test several times to be sure that your results are consistent.
Reproducible tests require a solid, documented methodology. When you're preparing to do a test, capture the configuration of all the devices involved as well as the applicable test scripts. By being methodical, you'll not only be able to efficiently repeat tests, but you'll also be able to adapt them to new equipment and test scenarios.
Done right, testing gives you accurate and defensible answers about your product purchases and deployments. You'll know ahead of time that a product is right for your network and that a patch or upgrade won't melt down your systems or compromise your security infrastructure. For a minimal cost of time and money, testing lets you check your deployment strategy ahead of time and minimize the risks associated with change.