We examined log files and configuration reports to determine how each appliance stacked up in enterprise management and control, daily operation, authentication and policies, and feature integration.
All of the products sell for between $12,000 and $18,500. But getting specific price configurations isn't easy, as each product has a complex range of user and feature licenses. Further confounding the pricing issue is that you will need to match the capacity of the product with the expected network traffic it will protect. We tried to compare appliances that had a similar number of network ports and capacity for a 1 Gbps external network connection.
We asked vendors to send us the boxes with the highest throughput possible and geared toward the largest networks. When we did our tests, we turned on all of the security modules--in the real world, this will severely limit their overall performance and is something to consider when deploying these products. However, we did not test performance. This is because testing performance is fraught with all sorts of issues. Either you test with synthetic clients to generate phony traffic so you can compare how different products respond on the "same" artificial lab network, or you do your tests on a live network and hope that the insights gained with your actual conditions are worth the loss of having the comparable traffic data. As a potential purchaser, you should match throughput specs with what you ultimately need on your network.
Configuration. All of the products, except Check Point, are primarily configured by connecting to their built-in Web servers. Check Point actually has three configuration interfaces--command line, a Web-based initial configuration tool for basic tasks, and its SmartView Monitor Windows-based administration tool (See Check Point screen shot, right). Unfortunately, you'll need to be familiar with all three. For example, you have to go to the command line interface to set up a DHCP server on an internal network.
Some of the Web interfaces are more logically designed than others. For example, Astaro, IBM ISS and Fortinet separate the functional modules--separate menu trees for antivirus and IDS, for example--and logically lay them out. Juniper has the poorest interface of the six, because its commands and controls are buried several levels down or require operators to visit multiple pages to set up even the simplest procedures, such as changing one of the antivirus settings. SonicWALL's interface is just a little better than Juniper, hiding many of its UTM features under a single "security services" menu tab.
Setting up the IBM ISS box took about an hour, and Check Point took several hours. The others were somewhere in between. While this may not be terribly important if you're installing a single box, it will add up for large deployments.
IBM ISS stood out from the pack with superior defaults, such as setting up internal network routes and activating features at the click of a button. This default-driven approach could be a bit problematic if your tastes run to doing something more sophisticated. For example, most of the other UTM appliances could handle connections to a WebDAV server for sharing files; with IBM ISS, we needed to set up a special firewall policy to allow this traffic. Nevertheless, this was a minor inconvenience--not enough to keep IBM ISS from getting the clear top grade in overall enterprise management.
Licensing, updating. Each product has intricate licensing and signature file update issues, mainly because customers will purchase varying configurations, feature sets and user counts. None of the products did a particularly good job troubleshooting licensing errors; Check Point and Juniper had the most complex and unintuitive licensing procedures. In fact, we had trouble with our Check Point licenses even after its engineer spent several hours on site setting up our box that turned up a bug. The other products make installing and upgrading licenses, and updating threat signatures, far easier.
IBM ISS makes this process a snap; it consolidates all of its updates for antivirus, IDS and firmware in a single screen. You can set it to check for updates automatically on a schedule. The others are more complex; you will have to visit multiple screens or do more than just push a single button to update everything.
Administration. Consolidated security administration is a key value proposition for UTM. However, getting to this consolidation won't be easy. Because these products cover a wide range of protection methods, they need to have the flexibility to be operated by multiple administrators.
Fortinet, Juniper and Astaro can handle multiple concurrent administrators and immediately post any configuration changes to their boxes in a "last one wins" scenario: This means that any intermediate changes will be ignored, which isn't ideal and means one person needs to have ultimate authority over all UTM appliances. Check Point, SonicWALL and IBM ISS only allow for a single administrator to be connected at any one time to avoid conflicts.
Check Point has the most complex and useful approach, providing great flexibility across a large deployment. Multiple administrators can run its SmartDashboard in read-only mode to view, but not change, the configuration. And it has other tools, such as the separately priced Provider-1, which can segregate roles between, say, a desktop department to handle antivirus configuration and a network group to manage the firewall setup. Juniper has something similar with its separately priced NetScreen Security Manager for managing role-based administration. (SonicWALL is coming out with a new version of its management software that will allow multiple concurrent admin users, but this wasn't available for our tests.)
Daily Operation
A critical value UTM products offer is the ability to quickly determine if your network has been breached or if you need to adjust the various protective mechanisms, since you have access to firewall, IDS and VPNs all in the same place. This means that if you mistakenly open a firewall port for the VPN, you can receive alerts to fix it without having to compare logs from two different places.
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
Juniper's quirky main menu (down the left side of this screen) presents control settings in almost random order.
|
 |
 |
 |
 |
 |
 |
 |
We used a typical scenario in which we ran the box for several days, examined the reports based on an initial firewall and protection rule set, and then adjusted our rules based on two situations--places where we wanted to eliminate false positives, and places where we needed to tighten down the box to prevent typical security weaknesses. Part of this exercise was to examine how reports would be created and examined and how threats will be evaluated and acted upon by the device.
Overall, Fortinet has the best set of tools to handle the day-to-day life of a security administrator, and Juniper scored lowest with its quirky main menu that scatters controls in almost random order (See Juniper screen shot, right). Juniper also requires that you visit several places to examine reports and other screens to change its protection rules. The other products are capable and about equal in this area.
Fortinet's front page gives you just enough details to monitor its overall operations. You can quickly find attack summaries in its menus, and the policy definitions are easy to set, and more importantly, easy to change when you have done something wrong.
Firewall-IDS. Part of the usefulness of a UTM appliance is how its firewall and IDS work together, and flexibility in terms of where it can be used across different configurations of an enterprise network. In other words, some products can position the IDS module outside of the firewall to repel attacks and reject this traffic before it is processed any further, or to work with an existing firewall infrastructure at a headquarters network.
Fortinet and Astaro can also examine incoming encrypted packet streams and act on this analysis before passing these streams through other modules, thereby saving on processing power.
Check Point, Juniper, Fortinet and Astaro IDSes scan for both attack signatures and attack behaviors. SonicWALL only analyzes behaviors and IBM ISS only signatures. The IDS modules of both IBM ISS and SonicWALL UTMs can also explicitly detect outbound attack signatures.
The SonicWALL, IBM ISS and Juniper IDSes are hard-wired to "live inside" the firewall, meaning that all network packets from the outside world go first to the firewall and then to the IDS for inspection. The advantage is that packets are filtered out by the firewall, reducing the inspection burden on the IDS. However, you do lose some insights because having the IDS outside the firewall can help you identify attack vectors early. This may be fine for organizations that manage both with the same administrative group, but problematic if the administrative roles are split.
Reporting. The products have varying methods for producing reports, with different levels of details. All of the vendors except Astaro sell separate reporting tools (not evaluated for this review) that work across their larger security product lines. This assumes that you have more than just UTM boxes from these vendors and want to consolidate reports so that all firewall information is in one place, all IDS alerts are in another, and so forth. This may not work for all usage scenarios, and could be cumbersome if you have multiple vendors' products in your data center. Having to purchase add-on reporting tools somewhat undercuts the purpose of having an integrated appliance.
Astaro includes reports as part of the Web administrative interface and produces an "executive report," which doesn't do much more than show some nice graphs of traffic flows.