Creating an antispam cocktail: Best spam detection and filtering techniques

While admins have a variety of weapons at their disposal in the fight against spam, no single weapon is capable of providing the ultimate death blow, killing all spam in its tracks. Fighting the daily torrent of spam, which (depending on who you ask) makes up 33-80% of all e-mail, requires the use of a "cocktail" approach, mixing multiple detection and filtering techniques.

An antispam strategy – whether that strategy is you or a vendor product – should incorporate three techniques, which complement each other to provide antispam defense where others fall short. Let's take a look at these three vital "ingredients" that should be part of any antispam cocktail.

1. Block mail from known sources of spam using lists of "bad" IP addresses compiled by companies or independent antispam crusaders. These lists are made up of addresses of systems and networks known to belong to spammers, so-called open relays and open proxies, which are poorly secured servers exploited by spammers and Web sites that host spammers or spammer support services. Two of the better known lists are SORBS – Spam and Open Relay Blocking System (http://www.us.sorbs.net/) and SpamHaus (http://www.spamhaus.org/).

   Freeware antispam cocktail One way to get a premixed version of the antispam cocktail is via the Apache Project's open source SpamAssassin, which combines all three antispam approaches. In addition to the cool moniker, SpamAssassin offers a good mixture of spam fighting techniques, clear documentation and active community support. And it's free (woohoo!). SpamAssassin is available at spamassassin.apache.org.

   Spam blocking lists are easy to use. Most modern e-mail servers can be configured to perform lookups on these lists via DNS queries with just a few lines of code in the configuration file. However, when you buy into a particular list, you trust the judgment of the list maintainers to determine who should be marked as a spammer.

2. Block mail based on content. Unless you are in the pharmacy business (or certain other businesses that I'll leave to your imagination), your company probably doesn't receive many legitimate e-mails containing the word "Viagra." Phrases such as "make money fast," "discount dvds" and "hot stocks" may also serve as a red flag for spam. By filtering mail containing these words, you can reduce the amount of clutter in your users' inboxes. Content filtering can also look for tell tale HTML that indicates the message is either spam or malicious content.

   There are two caveats here: First, the possibility of false positives, causing legitimate e-mail to be marked as spam. Second, the continuing ingenuity of spammers, who have taken to creative spelling techniques ("V1aGrA" or "V!agra"), use of html and graphics, as well as variations in spacing and punctuation to confuse and bypass such filters. This means that you'll need to keep adjusting your filters to deal with new types of spam as well as the spammers' new tricks to hide the true nature of their messages. If you are using a commercial anti spam product based on content filters, make sure that the vendor provides frequent filter updates.

3. Scientific content blocking. Bayesian filters use science, in the form of statistics, to identify spam. A Bayesian filter builds two tables -- one for all the words that appear in legitimate e-mails and the other for words that appear in spam -- and gives each word a score. At most companies, a word like "Viagra" is given a score that indicates that it is more likely to occur in a spam message than a word like "meeting." By looking at a message's overall "spamminess" score, the Bayesian filter can make a relatively accurate guess as to whether it is legitimate. The nice thing about these filters is that they can learn from e-mail over time. The more e-mails the filter scores, the better its scoring becomes.

   More information Go to E-mail Security School and earn CPE credits from (ISC)2. Take the e-mail security pop quiz. Visit our e-mail security resource center for the latest news, tips and advice on securing e-mail systems.

   Spammers have not stood still in the face of the Bayesian filters. You may have noticed strange blocks of text in some of the spam you receive. Spammers have taken to including passages of random, unusual or legitimate sounding text at the end of their e-mails in an effort to confuse Bayesian filters. The theory here is that by adding many words found in legitimate e-mail, the "spamminess" score of the message overall may be lowered.

   The learning ability of the Bayesian filter is also a double-edged sword. In order to make the best use of this technology, your users need to teach the system about spam messages that slip through the filter. While this is usually a simple point-and-click process, some users may be annoyed by this task or ignore it, thereby reducing the efficiency of the system.

As you can see, each of the ingredients in this antispam cocktail adds its own "kick" to the recipe. By combining and tuning these techniques, the savvy sys admin can reduce the level of spam on their network to a trickle instead of a torrent.

About the author
Al Berg, CISSP, CISM is the Director of Information Security for Liquidnet (www.liquidnet.com). Liquidnet is the leading electronic venue for institutional block equities trading. According to INC. magazine in 2004, Liquidnet was the fastest growing privately held financial services company in the US and the 4th fastest growing privately held company in the US across all industries.