Sizing Up E-mail Appliances

Is your organization inching along because of spam, malware and phishing attacks? We tested four e-mail appliances designed to clear the way for safe messaging. Here's how they measured up.

Spam is epidemic. An enterprise may typically receive 20,000 external e-mails per hour, 24 hours per day, and three-quarters...

of it's either junk or virus-infected. Winnowing this glut of bogus and often malicious messages while allowing legitimate business communication to flow is a huge challenge.

Think of an investment firm trying to filter spam offering the "the latest stock tip" without blocking advice to customers; or a hospital stemming the flow of "physique-enhancing pill" e-mails, but not risking missing a genuine patient inquiry.

Fortunately, e-mail security technology has advanced against increasingly sophisticated spamming techniques. Organizations have a wide choice of strong products and managed services to protect vital messaging.

More and more, the products are available in enterprise-class appliances to accommodate the staggering volume of e-mail flooding large organizations. We tested and evaluated four of the leading appliances: BorderWare Technologies' BorderWare MXtreme Mail Firewall, CipherTrust's CipherTrust IronMail Secure Platform, IronPort Systems' IronPort C-Series Email Security Appliance and Symantec's Symantec Mail Security (SMS) 8200 Series.

We discovered that any one of these appliances will do a highly capable job of protecting your organization against spam in addition to providing gateway AV protection--to the extent that this was not a differentiating factor based on our test results.

Accordingly, we evaluated these solutions on their appropriateness of installation, initial configuration, deployability/scalability, administration/management and reporting for a large enterprise.

About this BakeOff

Information Security tested four leading e-mail security appliances, focusing heavily on their antispam capabilities and overall suitability for a large enterprise environment.

Since our focus was not on processing power, the exact model was of little import, but, specifically, we tested BorderWare TechnologiesÕ MX 400 BorderWare MXtreme Mail Firewall, CipherTrustÕs CipherTrust IronMail Secure Platform on a standard IBM server, IronPort SystemsÕ IronPort C-60 Email Security Appliance and SymantecÕs Symantec Mail Security 8200.

Our test lab contained one server running Windows 2003, our SMTP and DNS servers, and Active Directory. Additionally, we had three workstations (Linux and Windows XP SP2) running on a Cisco network backbone. We used a mail generator/mail drain running on a BSD server.

For spam testing, we used the mail generator server to supply us with varying rates of live mail, and received the mail via the mail drain.

All of our testing was done inside a firewall on an isolated network (no outside-facing mail server, which limited our ability to test some third-generation features, such as reputation filtering).

The four vendors have some similarities and also some profound differences in how they filter spam. That being said, they all caught close to 100 percent of the spam we ran through them and registered no false positives. Our e-mail generating tool was a BSD server that simulated generating 1,000,000 messages to 10,000 recipients, allowing us to test the stability and scalability of the appliances. It also allowed us to see how the first- and second-generation spam filters would work under load. We were capable of creating both hard and soft bounces on the mail drain (receiver) as well, which meant that we didn't require a SMTP server for our testing and could conduct everything on an isolated network segment.

The differences in the products' antispam capabilities lie in third-generation technologies, which would be more likely to detect the more sophisticated techniques spammers use to evade detection. Since we used a closed lab environment, we didn't fully test these technologies.

Notably, IronPort and CipherTrust are on the leading edge of integrated antispam technology, primarily through their reputation filters. A reputation filter/service (identity-based filter) is used to analyze who is sending you mail and will block or delay messages based on the reputation of the e-mail source.

Though IronPort is credited with creating the technology, CipherTrust is attempting to push the envelope with its correlation engines. The IronPort reputation service, SenderBase, encompasses 75,000 networks and monitors 25 percent of all e-mail on the Internet. CipherTrust, whose TrustedSource network includes a somewhat smaller sampling of 3,000 enterprises, relies on a series of correlation engines to make inferences among different sources of e-mail.

Symantec's Brightmail BLOC (Brightmail Logistics Operations Center) service uses its patented Probe Network, which leverages millions of decoy e-mail accounts to capture spam. This information is then sent to BLOC, where a combination of automated tools and technicians determine if a message is spam. BLOC protects 15 percent of the world's e-mail--about 100 billion e-mail messages per month.

BorderWare's antispam technology comprises first- and second-generation tools, such as whitelists/blacklists, pattern matching and Bayesian filtering. The appliance we tested included the optional Brightmail engine. IronPort offers Brightmail as an add-on module, but the appliance we tested performed very well without it.

Antivirus A La Carte
Enterprises can typically see a 60 to 80 percent reduction in inbound e-mail-borne viral traffic when they use an AV gateway on their SMTP servers.

All the vendors--with the obvious exception of Symantec--use third-party AV technology: CipherTrust uses McAfee/Authentium; IronPort has Sophos; and BorderWare uses Kaspersky Lab and McAfee. IronPort also uses their proprietary Virus Outbreak Filters technology, which scans incoming mail for suspicious patterns that indicate possible zero-day malware attacks.

The AV engines detected everything we threw at them, including an EICAR file and "old" viruses--malware payloads currently living on the Internet. They also detected custom code with viral characteristics that we wrote in our isolated lab.

Antispam Techniques

In the ever-changing fight to block spam, here are some of the major detection techniques:

Whitelist/blacklist contains networks or domain names that are either trusted or untrusted to send legitimate e-mail.

Real-time blacklist (RBL) identifies servers that send out spam or are known to be open relays. (This definition is seen differently across various vendors.)

Message header tests interpret the contents of either the header or message body for valid IP addresses, sender address and destination addresses.

Bayesian filtering is a statistical approach to spam detection based on the probability of an individual word being used in a spam message. These filters must be constantly "trained" to understand variations of a word that a spammer may use. They can be circumvented by the use of HTML in the body of an e-mail message, though newer Bayesian filters are learning to combat this problem.

Machine learning typically consists of advanced statistical techniques run by an artificial intelligence engine to not only provide advanced e-mail text-based filtering (statistics), but to adjust or adapt itself based on the e-mail it's seeing (artificial intelligence).

Reputation filtering, based on very large samples of traffic, develops spam origination locations and blocks those locations.

Traffic shaping operates at the packet level. It not only looks at locations of spam senders, but actively controls the bandwidth allowed from those IP addresses to reduce the amount of mail it will even allow to be seen by the antispam appliance.w

--Tom Bowers

Enterprise products may not be plug-and-play, but one of the key benefits of an appliance is that it shouldn't require a lot of professional services or hours on the phone with tech support to get up and running. Our evaluation considered the documentation supplied and whether we could complete an installation without vendor professional services. If technical support was required, we evaluated their professionalism, courtesy and knowledge.

IronPort was clearly the smoothest installation, followed by CipherTrust. Symantec and BorderWare were somewhat problematic.

Once the appliances were installed, CipherTrust and IronPort presented the smoothest configuration experiences; the former because of its engineering support, the latter because of its documentation. On the other hand, they offer far wider sets of configuration options than either Symantec or BorderWare, meaning the experience may be more complex and time-consuming depending on how granular the appliance is tuned.

The IronPort installation starts with a command-line wizard and moves smoothly into a Web-based interface. A clear one-page installation check-off list and succinct Quick Start Guide helped us zip right through. The wizard was truly a fill-in-the-blanks installation. The only hiccup was at the end of the Web-based component, which failed due to cookies being disabled in Internet Explorer--an undocumented issue. We completed the install using Firefox (alternatively, we could have enabled cookies in IE). CipherTrust was the only vendor to insist on a telephone-supported installation, which went smoothly; our feeling was that we could have done well on our own. The manuals are nicely laid out, with plenty of screen shots and pictures of the appliance. Wiz-ards--though not quite as polished as IronPort's--for both the command line and browser interfaces guide you through the installation. However, the software update process is somewhat kludgy (you must query the update service for each update). CipherTrust's installation team was highly knowledgeable and very professional.

CipherTrust provides the most robust set of predefined filters, including antispam and antivirus settings based on industry best practices. Very few changes or modifications would be needed to configure your appliance for state-of-the-art e-mail protection. The only real challenge for CipherTrust is adding new rules, which requires a command-line style of coding. Wizards would make this a more complete package.

The BorderWare installation was more problematic. The appliance was sent without manuals; when we did receive the documentation, it was for a different model, but was close enough to get us through.

There were no pictures of the device to identify the network cards, the network adapters don't follow standard right-to-left or left-to-right conventions--0, 1, 2 or 2, 1, 0-- and only through three hours of trial-and-error testing to see if the box was live or dead did we figure out that the internal firewall drops ping packets. When the device responded to HTTPS, we finally determined which network adapter we had actually configured.

Once the command-line wizard is complete, you are left with a DOS-like menu that lacks a top menu bar to show possible menu selections (we accidentally pressed an arrow key and another menu box showed up). Neither the documentation nor online installation help gives any guidance on whether to choose an automatic or manual installation.

But, the license, software update and security processes are well thought out and straightforward. Setting up the box for mail was completed via a slick wizard. BorderWare provides a basic set of default rules, but changing settings is simply a matter of selecting check boxes.

The Symantec installation began with a three-day ordeal troubleshooting what turned out to be a bad box. The good news was Symantec's response: The company brought in any and all resources needed to remedy the situation, and it listened to its customer. Technicians dialed remotely into the appliance, and together we spent nearly six hours total over the three days trying to correct the problem. In the end, they sent a new box, which installed flawlessly.

This experience revealed two issues: Symantec's registration process needs to be more flexible and allow direct IP addressing. We learned this because the installation problem stemmed from the appliance's inability to use DNS to find Symantec's registration site, thus, we could not move into the browser-based interface to control the box. We had to use their command line, but Symantec admittedly "hides" the command-line commands available in the dark recesses of its documentation. The technical support folks stated flatly that they don't want customers using the command line.

Two other problems plagued this installation: The physical connectors are so poorly placed that we resorted to using old Cat 3 Ethernet cables (without rubber "hoods") to plug in; the same was true of the video cable. We had to "work" the video connector into the physical boundaries of the appliance.

On the plus side, the beauty of the Symantec configuration process is that, once you complete the command-line wizard, the basic mail settings are complete. This was a clear advantage over the other appliances. The default policies are very basic, but the intuitive wizard makes creating new policies easy.

Broadly speaking, our administration criteria incorporated the ease and flexibility of managing the devices and policies through the central console, particularly a policy/ filter creation/modification capability and the overall usability of the interface.

IronPort begins with a range of content filter options that include various header, attachment, MIME type and envelope scans. Each of these filter types can be broken down into smaller, logical comparative components as needed. A large array of possible actions can then be assigned to each of these filters. Its documentation provides numerous practical examples for implementing these filters.

While IronPort's default policy categories are limited (Whitelist, Blacklist, Suspectlist and Unknownlist), the policy options under each category are extensive, including maximum number of messages per session, banner test, max recipients per hour and using SenderBase. Each of these, in turn, is configurable to a remarkable degree: The SenderBase options alone fill 50 pages of its User Manual.

CipherTrust's granularity of policies and rules are as impressive as IronPort's, but its real jewel is the excellent best practices template it provides to set all of these filters right up front. You download the template with your initial updates, click apply, and the wide range of default settings are made--an astounding array of antispam, AV, content filtering and other settings. When you consider the 300 different settings available, this makes life a lot simpler for the harried security manager.

Symantec's default policies are the most basic. Its spam filter is set to prefix the e-mail header with the words "Suspected Spam" if it exceeds a specific default threshold. The mail filters are broken down into four basic categories--e-mail firewall, virus, spam and content compliance--but each of these subdivided into only two or three subcategories. For example, the e-mail firewall is broken down into directory harvest, spam and viral attacks.

While the base list of filters is minimal, customers will rely on the BLOC service to provide the granular filter for additional layers of protection.

BorderWare, which provides older antispam technology on its own with a Brightmail afterburner as an option, allows you to select basic filters such as whitelists/blacklists, RBLs, message header and envelope testing, Statistical Token Analysis (a form of Bayesian filtering) and not much else. The only real antispam configuration comes if you choose to add the optional Brightmail engine. You may enable the included proprietary secure Web mail portal--a nice option--which is unique among the four appliances.

BorderWare's really interesting options are in its HALO system managing, clustering, load balancing and stateful failover, including a number of policy thresholds designating the failover device.

Scaling for the Enterprise
We assessed the deployability of the appliances--the feature sets appropriate for enterprise deployment--including load balancing, clustering, failover and LDAP support.

All of the vendors support failover, load balancing and centralized management. However, only BorderWare natively provides clustering, failover and load balancing via its HALO programs.

The other vendors support external load balancing devices only, and they only support failover via MX record preferences. All four provide a central management console for support of a distributed multiple appliance deployment.

All but Symantec provide support for all industry standard LDAP directories. Symantec currently only supports Sun's Java Messaging System (formerly iPlanet) and Active Directory.

The bottom line is that any of these vendor solutions could support a global enterprise.


All of the appliances provide adequate levels of reporting and log access, though, overall. This was the most disappointing aspect of our evaluation, particularly for enterprise-caliber products.

CipherTrust was the best, providing easy access to both the appliance's log files and a wide range of reports. Its weakness is the lack of customization: You must export the logs files off and use an external report generator such as Crystal Reports. Additionally, the report export function is limited to CSV format.

IronPort was close behind, with a rich source of both logs and reports, though its logs were a challenge to extract. It also falls short in its customization and export functions.

Symantec provides a superb set of reports, but there is no customization. The logs are only fair and must be exported to a separate syslog server.

BorderWare offers only the bare minimum set of logs and only slightly better reports. It supports minimal customization, and you can only export reports via e-mail from the administrative interface.

All Viable Choices
Overall, we were impressed with these appliances. All of them would perform in a multiple-device deployment on a global scale. We also found them among the most secure devices we've seen in our labs, resisting all our efforts at compromise.

For innovation, we would certainly choose either the CipherTrust or IronPort solutions. Both vendors provide a rich mix of features, world-class antispam technologies and mature user interfaces. They get our best grades overall, with a nod to CipherTrust.

BorderWare is a viable choice for mid- to large-scale deployments because of its native load-balancing and failover, and the clustering option makes this solution easy to scale. Its use of third-party antispam and AV technology makes it a predictable performer.

Symantec is a tried-and-true option for deployment. Innovation is not its strong point, but its global presence and support structure are definite strong suits.

All in all, this is a mature product group with a strong future as spam, phishing and virus attacks continue to grow.

Technical editor Tom Bowers, CISSP, PMP, CEH, is a manager of information security operations at a Fortune 100 pharmaceutical company. Send your comments on this article to
This was first published in August 2005

Dig Deeper on Email Security Guidelines, Encryption and Appliances



Find more PRO+ content and other member only offers, here.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: