Product review: Unified threat management (UTM) devices

Unified threat management devices consolidate several network security functions into one product. This article evalutes six UTM appliances; each had to act as a firewall and virtual private network and provide antivirus, Web content filtering, intrusion prevention and antispam protection.

Is your business ready To roll network security into a single platform? We evaluate six leading UTM appliances...

to help you push the right buttons.

Unified Threat Management is a growing, competitive field, with more than a dozen vendors. The idea is to consolidate your security appliances into a single box and manage an integrated protection profile for your corporate network. While especially appealing for companies with several branch offices without any resident IT or security staff, the implementation isn't perfect for a corporate-wide deployment, primarily because of limits on how you are allowed to administer the component security applications.

We asked vendors to deliver a product that could act as a firewall and virtual private network gateway, and protect our test network against attack with a minimum of four defense mechanisms--antivirus, Web content filtering, intrusion prevention and antispam.

We reviewed six UTM appliances in a head-to-head evaluation: Astaro Internet Security's Astaro Security Gateway 320; Check Point Software's UTM-1 2050; Fortinet's FortiGate-1000A; IBM Internet Security Systems' Proventia Network Multifunction Security MX5010; Juniper Networks' SSG 550 and SonicWALL's SonicWALL Pro 5060c.

We examined log files and configuration reports to determine how each appliance stacked up in enterprise management and control, daily operation, authentication and policies, and feature integration.

All of the products sell for between $12,000 and $18,500. But getting specific price configurations isn't easy, as each product has a complex range of user and feature licenses. Further confounding the pricing issue is that you will need to match the capacity of the product with the expected network traffic it will protect. We tried to compare appliances that had a similar number of network ports and capacity for a 1 Gbps external network connection.

We asked vendors to send us the boxes with the highest throughput possible and geared toward the largest networks. When we did our tests, we turned on all of the security modules--in the real world, this will severely limit their overall performance and is something to consider when deploying these products. However, we did not test performance. This is because testing performance is fraught with all sorts of issues. Either you test with synthetic clients to generate phony traffic so you can compare how different products respond on the "same" artificial lab network, or you do your tests on a live network and hope that the insights gained with your actual conditions are worth the loss of having the comparable traffic data. As a potential purchaser, you should match throughput specs with what you ultimately need on your network.

About this Review

We connected each UTM box on a test network with Windows XP, Vista and Apple Macintosh clients and a Windows 2003 Enterprise Server running Microsoft's IIS Web server.

Each UTM box was configured with two interfaces--a local network with a DHCP server enabled, and an external network connecting to our DSL modem. We set up firewall and intrusion rule sets, ran Outlook Express POP email clients, and used Skype, GoogleTalk and AIM messaging sessions.

We also connected to a WebDAV server to share files over the Internet. We connected to each product's built-in Web management server using both Firefox v2 and Internet Explorer 6 and 7. We also used SSH to perform command-line configuration tasks when necessary.

--David Strom

Enterprise Management and Control

We examined how each product is managed, typically with a Web browser, and the various administrative roles that can be performed concurrently. We also looked at how particular functions are licensed, and how threat signatures are updated. Because these products handle a variety of security tasks, ease of setup is important, and being able to delegate and divide administrative roles is also critical.

Check Point's SmartView Monitor for administering its operations

Configuration. All of the products, except Check Point, are primarily configured by connecting to their built-in Web servers. Check Point actually has three configuration interfaces--command line, a Web-based initial configuration tool for basic tasks, and its SmartView Monitor Windows-based administration tool (See Check Point screen shot, right). Unfortunately, you'll need to be familiar with all three. For example, you have to go to the command line interface to set up a DHCP server on an internal network.

Some of the Web interfaces are more logically designed than others. For example, Astaro, IBM ISS and Fortinet separate the functional modules--separate menu trees for antivirus and IDS, for example--and logically lay them out. Juniper has the poorest interface of the six, because its commands and controls are buried several levels down or require operators to visit multiple pages to set up even the simplest procedures, such as changing one of the antivirus settings. SonicWALL's interface is just a little better than Juniper, hiding many of its UTM features under a single "security services" menu tab.

Setting up the IBM ISS box took about an hour, and Check Point took several hours. The others were somewhere in between. While this may not be terribly important if you're installing a single box, it will add up for large deployments.

IBM ISS stood out from the pack with superior defaults, such as setting up internal network routes and activating features at the click of a button. This default-driven approach could be a bit problematic if your tastes run to doing something more sophisticated. For example, most of the other UTM appliances could handle connections to a WebDAV server for sharing files; with IBM ISS, we needed to set up a special firewall policy to allow this traffic. Nevertheless, this was a minor inconvenience--not enough to keep IBM ISS from getting the clear top grade in overall enterprise management.

Licensing, updating. Each product has intricate licensing and signature file update issues, mainly because customers will purchase varying configurations, feature sets and user counts. None of the products did a particularly good job troubleshooting licensing errors; Check Point and Juniper had the most complex and unintuitive licensing procedures. In fact, we had trouble with our Check Point licenses even after its engineer spent several hours on site setting up our box that turned up a bug. The other products make installing and upgrading licenses, and updating threat signatures, far easier.

IBM ISS makes this process a snap; it consolidates all of its updates for antivirus, IDS and firmware in a single screen. You can set it to check for updates automatically on a schedule. The others are more complex; you will have to visit multiple screens or do more than just push a single button to update everything.

Administration. Consolidated security administration is a key value proposition for UTM. However, getting to this consolidation won't be easy. Because these products cover a wide range of protection methods, they need to have the flexibility to be operated by multiple administrators.

Fortinet, Juniper and Astaro can handle multiple concurrent administrators and immediately post any configuration changes to their boxes in a "last one wins" scenario: This means that any intermediate changes will be ignored, which isn't ideal and means one person needs to have ultimate authority over all UTM appliances. Check Point, SonicWALL and IBM ISS only allow for a single administrator to be connected at any one time to avoid conflicts.

Check Point has the most complex and useful approach, providing great flexibility across a large deployment. Multiple administrators can run its SmartDashboard in read-only mode to view, but not change, the configuration. And it has other tools, such as the separately priced Provider-1, which can segregate roles between, say, a desktop department to handle antivirus configuration and a network group to manage the firewall setup. Juniper has something similar with its separately priced NetScreen Security Manager for managing role-based administration. (SonicWALL is coming out with a new version of its management software that will allow multiple concurrent admin users, but this wasn't available for our tests.)

Daily Operation
A critical value UTM products offer is the ability to quickly determine if your network has been breached or if you need to adjust the various protective mechanisms, since you have access to firewall, IDS and VPNs all in the same place. This means that if you mistakenly open a firewall port for the VPN, you can receive alerts to fix it without having to compare logs from two different places.

We used a typical scenario in which we ran the box for several days, examined the reports based on an initial firewall and protection rule set, and then adjusted our rules based on two situations--places where we wanted to eliminate false positives, and places where we needed to tighten down the box to prevent typical security weaknesses. Part of this exercise was to examine how reports would be created and examined and how threats will be evaluated and acted upon by the device.

Juniper's quirky main menu (down the left side of this screen) presents control settings in almost random order.

Overall, Fortinet has the best set of tools to handle the day-to-day life of a security administrator, and Juniper scored lowest with its quirky main menu that scatters controls in almost random order (See Juniper screen shot, right). Juniper also requires that you visit several places to examine reports and other screens to change its protection rules. The other products are capable and about equal in this area.

Fortinet's front page gives you just enough details to monitor its overall operations. You can quickly find attack summaries in its menus, and the policy definitions are easy to set, and more importantly, easy to change when you have done something wrong.

Firewall-IDS. Part of the usefulness of a UTM appliance is how its firewall and IDS work together, and flexibility in terms of where it can be used across different configurations of an enterprise network. In other words, some products can position the IDS module outside of the firewall to repel attacks and reject this traffic before it is processed any further, or to work with an existing firewall infrastructure at a headquarters network.

Fortinet and Astaro can also examine incoming encrypted packet streams and act on this analysis before passing these streams through other modules, thereby saving on processing power.

Check Point, Juniper, Fortinet and Astaro IDSes scan for both attack signatures and attack behaviors. SonicWALL only analyzes behaviors and IBM ISS only signatures. The IDS modules of both IBM ISS and SonicWALL UTMs can also explicitly detect outbound attack signatures.

The SonicWALL, IBM ISS and Juniper IDSes are hard-wired to "live inside" the firewall, meaning that all network packets from the outside world go first to the firewall and then to the IDS for inspection. The advantage is that packets are filtered out by the firewall, reducing the inspection burden on the IDS. However, you do lose some insights because having the IDS outside the firewall can help you identify attack vectors early. This may be fine for organizations that manage both with the same administrative group, but problematic if the administrative roles are split.

Reporting. The products have varying methods for producing reports, with different levels of details. All of the vendors except Astaro sell separate reporting tools (not evaluated for this review) that work across their larger security product lines. This assumes that you have more than just UTM boxes from these vendors and want to consolidate reports so that all firewall information is in one place, all IDS alerts are in another, and so forth. This may not work for all usage scenarios, and could be cumbersome if you have multiple vendors' products in your data center. Having to purchase add-on reporting tools somewhat undercuts the purpose of having an integrated appliance.

Astaro includes reports as part of the Web administrative interface and produces an "executive report," which doesn't do much more than show some nice graphs of traffic flows.


Live monitoring. We examined several critical pieces of information available from the Web interface: real-time CPU and memory load, current alerts of potential network attacks, antivirus-related messages, and system health messages that required immediate attention.

This is helpful to see if your UTM box is overloaded or mismatched with the particular network traffic and inspection loads.

All of the UTM products except Check Point and IBM show the current CPU load and, in some cases, memory consumption on the home page of their Web interface, so it is easy to find and easier still to track. IBM ISS buries its status screen, while you have to visit Check Point's SmartView Monitor (a separate piece of software that comes as part of the UTM package) to get this information.

The three most useful front pages were from Astaro, SonicWALL and Fortinet, which offer all sorts of helpful summary information in one convenient place. Fortinet also includes a secure command-line console window within its Web interface, while the others require an SSH client to connect to their box if you need access to the command line. SonicWALL also tells you if you have set up the box with a known security weakness, such as allowing management from the WAN interface.

Check Point uses Windows software for its management, which means an admin must carry around a laptop with the software installed, rather than simply logging in through a browser. IBM ISS and Astaro can't be managed through Macintosh-based Firefox browsers, and we found some bugs when we administered SonicWALL with Firefox on a Mac.

IBM ISS' antivirus status screen shows protocols protected and traffic statistics.

Antivirus statistics are very important, since few things light up the help desk lines like email problems. IBM ISS has a simple-to-understand antivirus status screen (See IBM ISS screen shot, right), showing messages blocked, signatures, and which ports are being blocked or scanned. Astaro also has a good summary display of its email traffic, but tweaking the protection results requires visiting several different sub-menus. Check Point and Fortinet put this information on summary screens; Juniper and SonicWALL have separate screens that summarize the virus penetrations.

Authentication and Policies

Setting up and tuning security policies for the various modules is at the core of these products. Ideally, you would want an appliance that makes it easy to figure out how to keep your network protected, but still allows users room to get actual work done, all the while providing feedback when you have too strong or too weak a policy.

SonicWALL and Fortinet clearly lead the pack in this regard with the others scoring equally behind. Even if you don't activate all of the security modules, both vendors' approach is easy to understand and provides just enough feedback so as to not overwhelm an administrator.

Fortinet protection profiles provide a good base that can be modified for particular requirements.

There are two basic approaches to how security policies are created:

  • Integrated policy that applies to particular users or network interfaces. This has its advantages if your UTM box sits on several different network segments and you want to deploy different policies by segment or by user group (for example, one with servers on it, or one with engineering users). With this method, an administrator sets one policy that cuts across all of the individual security modules, with specifics for antivirus, IDS and so forth. Call this the traditional firewall approach, and each policy can enable different security modules for particular situations.

    Fortinet and Check Point use this approach; Fortinet does a better job, setting up a series of four default protection policies that gives you a great starting point and examples that make it easy to modify them for your specific needs (See Fortinet screen shot, right).
  • Separate policies that are module-specific. This means there will be one policy for antivirus, another for general firewall tasks, and more for IDS actions. IBM ISS uses this approach; while it also has chosen lots of defaults to get you started, making modifications isn't as easy as with Fortinet, because you must make them in several places. Juniper also sets up security policies by module.

The appropriateness for your company depends largely on how you have structured your support staff. If you have an antivirus person on staff, and you have a box that requires adjusting antivirus policies in several different places, you have a lot more maintenance work than with a box where you can set these policies in a single place. However, your security staff may wear a lot of different hats and thus this might not be as much of an issue. It is really a matter of taste and organizational structure.

SonicWALL zones offer module-specific protection policies.

SonicWALL and Astaro mix both approaches. Astaro has policies that are based on application-layer protocols (Web, email, IM and so forth) and has separate policies for network layer events. This means that to make changes in the UTM operations, you need to touch screens in both the protocol section and the network interfaces. If you forget one or the other, you will have configuration problems or, worse yet, think you are protected when you aren't.

SonicWALL policies are module-specific, and are applied to particular network routes. That has a lot of appeal, and is why we give it top marks here. All of its protection rules are organized in a single section, and it is easy to apply them to the appropriate interface (See SonicWALL screen shot, right).

Authentication capabilities are relevant if they are used for remote VPN connections. For most site-to-site VPNs, this isn't important unless you want to do some rudimentary endpoint protection or create policies based on particular user groups or roles.

All of the products support RADIUS authentication; Astaro and Fortinet can connect directly with Active Direc-tory user store; Astaro also supports authenticating to Novell's eDirectory. Juniper can integrate with RSA's SecurID tokens directly.

All of the products offer IPsec VPNs, and Astaro, Check Point and Fortinet support SSL VPN terminations. None of the SSL modules has anywhere close to the level of features that a standalone SSL VPN box would provide.

Feature Integration

Our final series of tests looked at how the various functional modules work together. We also determined the third-party suppliers for these modules and what noteworthy features one product has that the others do not.

We gave SonicWALL the top grade because of its superior antivirus features, protection rule flexibility and implementation of IM protection across all of its security modules. Juniper and IBM ISS scored lowest because of the difficulty in making changes to their protection rules. For example, in order to implement protection or blocking of a specific protocol, you have to hunt down the rules that apply to that protocol and make adjustments in several places in the user interface. The other products fall somewhere in between in terms of complexity.

Each product uses different combinations of home-grown and third-party security services to round out its UTM coverage. Astaro, Check Point and Juniper use SurfControl for Web content filtering, while the others have developed their own content-filtering capabilities. Astaro uses Snort, while the others have their own IDS engines.

Astaro supplies three virus scanners--a proprietary one using the Authentium antivirus engine, another based on open-source Clam AntiVirus, and a PCI hardware-based antivirus capability from Sensory Networks. Juniper uses Kaspersky, and Check Point uses CA. IBM ISS uses Sophos, along with a second scanning algorithm that examines network behavior. SonicWALL and Fortinet have their own antivirus scanners.

The six products differ on how big a file attachment they will scan through their antivirus engines. SonicWALL claims an unlimited file size because it scans while streaming the packets, while the others are more limiting because they have to cache the files first. If performance bogs down, an administrator can automatically block files beyond a certain size. IBM ISS hides this setting in its advanced settings, while the others make it easier to adjust the maximum limit.

Astaro presents a wide variety of choices to allow, monitor or block IM sessions.

All of the products can at least monitor IM traffic (See Astaro screen shot, right, for example), and some have rudimentary mechanisms to (sometimes) block particular IM protocols. SonicWALL was the only vendor that can completely block Google Talk and Skype conversations. Fortinet's IM protection is somewhat obscure. You have to go to two different places, one to handle policies for individual users and one to monitor or block the specific IM protocols.

Any solid defense against IM use will require combining Web filters to block access to particular sites as well as using the IM modules' features.

Check Point also does some very extensive port scanning, including ports that are used for VOIP, IM and P2P applications.

Web application scanning is absolutely essential if your company's Web servers are in remote locations or if you plan to set up a new Web server on an unprotected network such as at a branch office. Check Point, SonicWALL and Juniper offer protective mechanisms for preventing common Web application attacks such as SQL injection and cross-site scripting. We didn't find policy setting particularly straightforward for any of them.

The others just give lip service here, or require you to spend your days writing firewall rule sets.

For additional features, we liked Check Point's safe upgrade, requiring an administrator to complete a successful login within a specified (and user-selected) period of time; otherwise the box will roll back to a previous version. SonicWALL allows management of its wireless access points from its UTM device.

Not All Things to All People

UTM is one of those concepts that sounds great in theory, but is messy in practice. The six products tested all had their quirks, and we would have found show-stopping issues on all of the boxes if we didn't have a lot of support from each vendor.

While Fortinet and SonicWALL clearly have the best collection of features and Juniper the weakest, the others all had their good points, and the differences among each of the products is more a matter of taste and judgment than anything else.

Weigh the ability for multiple people to manage these boxes with how you organize your security staff. If you have separate groups managing firewalls and antivirus, for example, you might be better off choosing the products that separate their security policies.

You will also want to examine how a UTM deployment for your branch offices--which makes a lot of sense and can reduce your overall support burden--will be balanced with the products that you use or will use on your headquarters network. While Check Point and Juniper have solid solutions for the headquarters, they have less satisfying and less mature UTM product lines. Think carefully about what functions and modules you want to consolidate, and how you will go about managing the appliances before you invest heavily in any solution.

Report Card

Click here to see how these six leading UTM appliances make the grade (PDF).

This was last published in June 2007

Dig Deeper on Network Intrusion Detection (IDS)



Find more PRO+ content and other member only offers, here.

Join the conversation

1 comment

Send me notifications when other members comment.

By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy

Please create a username to comment.

We've recently purchase a FortiGate 100D UTM appliance, and are using it with version 5 of its FortiOS software.
We find the documentation to be too-long, confusingly structured, inaccurate both technically in describing concepts and practically in that the documentation sometimes mismatches how the FortiGate device and its FortiOS software actually work, as well as buggy (or some might say that it has curious "features").

Examples include insistence in the docs that a magical "ssl.root" pseudo-interface must exist in policies in order for VPN traffic to be able to transit the appliance. But I have no reference whatsoever to ssl.root and my VPNs work. They also NAT, even though I can't find where that NAT is enabled...

The process for installing a proper CA-issued SSL certificate for the SSL VPN is documented ... but neither where nor how you expect, leading to lost time until we found a forum post explaining how to do it.

The FortiClient (for Windows, MacOSX, Linux and Android) is NOT one client, but rather three or four separate products, each with different features/capabilities, limitations, usability and bugs. (iOS has no FortiClient, can only access FortiGate VPN via IPsec, and requires a completely separate VPN server configuration on the FortiGate appliance from the other VPN client types).

We have encountered weird routing issues with a completely disconnected "Management" interface on the appliance, when we connect IPsec VPN clients. Other users have confirmed it; FortiNet has reproduced it ... and there is neither explanation nor answer (although we do know the workaround - make sure that the Management interface’s IP address configuration is cleared, itself requiring the unintuitive assignment, as you cannot leave the field simply blank).

The FortiToken service can be broken (as in, made to not work correctly). FortiNet insists on putting its token configuration and licensing servers between your FortiGate appliance and the 2-factor authentication client on your users' endpoints, instead of enabling general OAUTH compliant tokens to be serviced directly from the FortiGate appliance. This is a(n artificial) revenue stream for FortiNet, and an operational pain for you (refer back to "I broke it" - I now have a FortiToken which cannot be used, associated with a user configured on the FortiGate appliance which cannot be edited while that token is associated; FortiNet support has been working on this for a week so far without positive result).

The Firewall Policy user interface on the FortiGate appliance blends conventional firewall policies (from interface, from source IP address ranges, to interface, to destination IP address ranges, date/time, services, permissions and whether to NAT) with VPN specific policies which include authentication and access magic, in a single table-style user interface. The column names for both conventional and VPN policies are “From” and “To”, but they mean different things in the context of a conventional Firewall policy statement versus a FortiNet VPN specific policy statement. And the summary shown in the table display of the policies hides much critical detail, making it easy to misunderstand the rules as they already exist.

The use of user authentication groups to associate a user with a particular SSL VPN “Portal” (which is not a VPN, it’s an SSL protected web page which offers portal functionality, optionally also with proxies; but it also is the way that SSL VPN real VPN connections are set up) can cause non-deterministic results depending on the order in which the user’s group affiliations are enumerated and the order in which those are checked against the available SSL VPN portal configurations. This can cause some strange behaviors (obviously valid user login attempts getting errors) and requires excess caution in configuration and in ordering certain policy statements.

And more.
In short, for such a mature product, it is unnecessarily opaque to configure, surprisingly poorly documented for all of its verbosity, and buggy.

I imagine that after absorbing the frustrations which come out of these things, and the significantly larger than should-have-been-necessary amount of time to learn the FortiNet way of doing things (plus the workarounds for incorrect docs, software bugs, etc), that it will prove to be a highly functional and very reasonably priced solution. But I had expected much better.