One thing I've learned from testing the latest crop of firewalls: Don't believe anything the vendors claim until...
you've tested it yourself. This means things that "ought to work" or "used to work" may not work at all, or may not behave as you expect.
In this tip, I will discuss how to test a firewall, including the three types of firewall tests you should perform and the types that surprisingly aren't necessary, to ensure you choose the best firewall for your organization.
The process for testing firewalls can be broken down into three distinct phases: subjective evaluation, efficacy (effectiveness) of threat mitigation and performance testing.
Testing firewalls: Subjective evaluation
Your subjective evaluation should be based on a criteria list, not a feature list. Look at each part of the firewall, such as how rules are defined, how VPNs are built, how remote access works, and how threat mitigation is layered on top of the product. Document your findings and accompany your notes with lots of screenshots. Otherwise, what seems obvious when you're testing Firewall A will be confusing when you revisit those findings while looking at Firewall G six weeks later. Take good notes on each criterion you evaluated.
When you have evaluated all of the firewalls under consideration, do a "horizontal sweep" that summarizes all the products for each criterion. That will make it easy to assign scores and draw an unbiased conclusion.
Testing firewalls: Efficacy testing
Efficacy testing is hard to do without specialized tools, and even if you have specialized tools, you may not get good results. Efficacy testing should focus on three areas: intrusion prevention, antimalware and application identification.
For intrusion prevention system (IPS) testing, my company uses products from Mu Dynamics Inc., although other test vendors such as Spirent Communications plc and IXIA Inc. have similar products. You can buy or rent these tools if you want, but you should be able to get each firewall vendor to run the tests you specify as they usually have the same tools.
For antimalware and application identification testing, you can't do an efficacy test. Instead, spot-check by testing scenarios with real clients downloading real viruses or communicating with real applications. One nice source for discovering recent viruses is the quarantine on your mail gateway. If your endpoint antivirus software quarantines viruses in a central place, that's another good place to check. When testing antivirus efficacy, try your own evasion: HTTP and HTTPS on non-standard ports, encrypted SMTP and IMAP, and proxy tunneling are all good types of evasion to add to the normal HTTP/HTTPS tests.
For application identification testing, pick the applications you care about most and test against real servers. If you want to block peer-to-peer file sharing, launch a few different Torrent clients and see what happens. Do the same for applications such as webmail or Facebook, which are both prime candidates for application identification and control. Don't try an automated test tool, as the results are never as accurate as the real application talking to real servers. This is especially true of applications that are evasive, such as BitTorrent and Skype, which can never be perfectly simulated in a test tool.
Testing firewalls: Evaluating performance
Performance testing also usually requires specialized tools, but has become so popular that there are open source alternatives. When testing performance, remember to check your test bed against a null device; a router or patch cable would work. This will tell you the maximum speed of your test bed. From there, keep in mind noted network tester David Newman's Laws of Testing: It must be repeatable, it must be stressful (to the device, not to you!), and it must be meaningful. Take the device you're testing to its limits, even if you don't anticipate going that far. This will tell you where you will hit a wall in the future and where you have adequate headroom to grow.
Don't performance test a thousand different scenarios because your network only has one scenario: reality. Try to build a small set of scenarios that represent your network and your usage, and test against an equally small set of configurations of the firewalls. Since most firewalls do most of their work when handling HTTP and HTTPS traffic, you can safely focus on that. Adding in that extra 3% or so of DNS makes things a lot more complicated and usually won't tell you anything useful.
Performance testing has to be accompanied by "pass/fail" indicators. For example, when the firewall starts to refuse to open new sessions, the test should end as you have gone beyond the limits. You should also set other limits, such as maximum latency time, to define when the firewall is not behaving acceptably.
With these three types of tests completed, you will have a solid understanding of best firewall products for your organization.
About the author:
Joel Snyder is a senior partner with consulting firm Opus One in Tucson, Ariz.