This article can also be found in the Premium Editorial Download "Information Security magazine: Betting the house on network anomaly detection systems."
Download it now to read this article plus other related content.
|For Tests, Go Virtual|
Your permanent test lab should run on bare hardware; for everything else there's EMC's VMware.
This gem lets you run multiple virtual machines on a single server. It's also a security tester's best friend since it lets you quickly deploy a "clean" system for testing purposes. Having VMware images of multiple Windows, Linux and FreeBSD versions allows you, for example, to deploy a clean Windows Server 2003 — patched and preloaded with some common utilities — in just a few minutes.
VMware has become the de facto standard for QA testers around the world, and anyone seriously taking on the problem of patch management has already built a few servers to help in that task.
Although EMC would prefer that you buy the VMware GSX Server, you'll find that the additional capabilities of GSX Server (such as multiple remote managers) are only necessary on one or two of the largest and most heavily used servers. You can often manage the rest using the more affordable VMware workstation product.
Following its acquisition of Connec-tix, Microsoft released its own virtualization products — Virtual PC and Virtual Server 2005. As with VMware, the main difference between the two is management capabilities. In a production environment, you'll definitely want to go with Virtual Server 2005, which offers both stronger management and a data center focus. However, for the test lab, Virtual PC offers the core capabilities you need at a lower cost.
Many testers have a warm place in their hearts for Symantec's Ghost, which allows them to quickly build and deploy servers with virtually any OS on Intel hardware. Although Symantec has taken Ghost far into the world of system management and deployment for production systems, many of the bells and whistles added in recent versions don't add much for test lab functionality.
— JOEL SNYDER
Assemble a Toolkit
A test lab is invaluable as a sandbox for testing new configurations, interoperability and patches, and answering the two main questions plaguing every security manager: Does it work, and is it fast enough?
The open-source world has a wealth of outstanding testing tools. Your starting point should include applications such as Nessus (vulnerability scanner), Nmap (port scanner and system profiler), Netcat (TCP and UDP pipe tool), ping and traceroute. These products are well-documented and very reliable.
Beyond these basics, you'll want to branch out into more specific tools. In each category, you can find open-source tools of varying quality and usefulness to test coverage and performance of security products. The important thing to remember when using open-source tools is that they are often built to examine specific items. For example, several packages, such as "stick," focus on IDS evasion--a widely discussed issue largely irrelevant to most enterprise IDS deployments.
Security testing tools are designed to run on a mix of Windows and Unix systems. Laptops are great for basic functionality testing, especially Apple's OS X models; they give you easy access to standard tools such as Microsoft Office and Adobe Acrobat, yet have the Unix underpinnings required by most open-source tools. However, you'll find that having a few Windows-based laptops is critical, as many security products have Windows-only GUIs or require Internet Explorer.
At the server level, don't underestimate the number of systems you'll need. While some sort of virtualization (EMC's VMware or Microsoft's Virtual PC) should be one of your very first purchases, there's no substitute for a variety of different systems when it comes to running tests (see "For Tests, Go Virtual," p. 41).
Once you have your coverage and functionality test equipment set up, you'll want to dive into the world of performance testing, driving the hardware and software until it collapses under load. Your choice of equipment depends on what you need to test, but most test labs will want to start with Layer 2 and Layer 3 packet generation tools and Layer 7 application stress tools.
Layer 2 and Layer 3 packet generators--such as Spirent's SmartBits--create flows at virtually any speed. They'll generate the background noise that puts stress on the equipment you're testing so your real test tools can do their work. For example, sending a well-crafted attack through an IPS is interesting at one attack per minute, but it's a lot more interesting if the IPS sees the attack when it's simultaneously being pummeled by 100 Mbps of non-attack traffic. Layer 3 packet generators are very useful in testing VPN equipment by putting stress on the crypto engine and verifying the advertised capacity.
Layer 7 test tools, such as Spirent's Avalanche and Reflector, generate real application-layer streams over TCP and UDP. These are important because you'll often test equipment that looks at and modifies the data as it flows through the network.
Make It Repeatable
The most difficult--and important--part of testing is the development of a standard test plan. David Newman of Network Test, a benchmarking and network design consultancy, has three rules that form the game plan for anyone testing systems: Tests must be meaningful, stressful and repeatable.
Focus on what's meaningful--you need to inventory and survey your network to find out what is really going on. If you're testing firewalls, you need to know what protocols are flowing through your network, how many connections per second occur, how many octets flow over each connection, how many connections are open at once, and how complex your rule set is.
Settings such as logging can also have a huge effect on performance. Vendor tests are always done with logging turned off, but you'll want to test with logging turned on if your policy is to log everything to a SIM. As you gather this kind of data, keep a log of the different information.
Testing must stress the systems to their real configuration and performance limits. Knowing that a firewall can cope with existing load isn't interesting, but knowing how much your load can grow before the firewall collapses is invaluable. Stress isn't just a performance issue. If you have a 500-rule firewall, see what happens when you put 5,000 rules in it.
Can such a configuration be supported? If you're testing a high-availability system, how does it handle different failure scenarios? Don't just look at the ones that seem likely--like sudden power cord attenuation--but more stressful failures, like a network cable disconnecting and reconnecting several times a second for a few hours.
Determine when a system has failed. Most security devices have gradual system degradation rather than a hockey-stick plunge in the performance curve. As part of your test plan, set metrics ahead of time that define when a system is no longer acceptable. For example, a firewall that adds 2 milliseconds of latency is acceptable, 20 milliseconds is marginal and 200 milliseconds is unacceptable.
Finally, your tests have to be repeatable. You'll need to repeat the results in your own lab as vendors change product versions and as you make head-to-head comparisons. And, of course, you have to run every test several times to be sure that your results are consistent.
Reproducible tests require a solid, documented methodology. When you're preparing to do a test, capture the configuration of all the devices involved as well as the applicable test scripts. By being methodical, you'll not only be able to efficiently repeat tests, but you'll also be able to adapt them to new equipment and test scenarios.
Done right, testing gives you accurate and defensible answers about your product purchases and deployments. You'll know ahead of time that a product is right for your network and that a patch or upgrade won't melt down your systems or compromise your security infrastructure. For a minimal cost of time and money, testing lets you check your deployment strategy ahead of time and minimize the risks associated with change.
This was first published in July 2005