Tip

Reputation systems gaining credibility in fight against spam

What's in a reputation? In a lot of cases, it can mean the difference between being overrun by spam, phishing and other undesired email, and keeping your users' inboxes free of potentially dangerous payloads. Spam has undergone a radical evolution during the past few years, and reputation systems are now a key technology in dealing with the ever-increasing volume of unwanted messages.

Historically, spam has been detected using a set of signature-matching and heuristic approaches optimized using a weighting system. This worked fine for a while, but as volumes continued to increase exponentially, spam gateways and services couldn't keep up. It was too resource intensive to conduct a detailed scan on every message.

    Requires Free Membership to View

Listen to Mike Rothman's tip

This tip is also featured as this week's Threat Monitor podcast. Download his spam prevention advice to your PC or favorite MP3 player.
Lately, the problem has been become more acute due to the wave of image spam. These messages use randomly sliced and pixilated images to evade spam detection. Many vendors have responded with optical character recognition (OCR)-based approaches to interpret words in the images. But OCR is resource intensive, further impacting the scalability of current anti-spam gateways.

Enter reputation systems, stage left. Reputation systems have been in use for the past three years, but are only now becoming "table stakes" for any vendor offering email security solutions. That is, it's hard for any vendor to substantiate a high spam detection rate without relying on reputation.

The general concept behind a reputation system is that you can, with some precision, figure out the likelihood of a message being spam, based on who is sending it. That's right: based upon the IP address of the sender, it has become possible to determine the sender's intent.

Why not use the sender's address -- or some other attribute of the message -- to assess reputation? Basically because IP addresses cannot be spoofed; they identify the sender and receiver of an email message and are essential to ensuring a message gets to its destination. You can fake pretty much everything else about a message, but not the originating IP address.

In order for a reputation system to be effective, the reputation provider needs to see a bunch of data -- think billions of messages -- in order to conduct a comprehensive analysis that will yield accurate results. Look for vendors that have access to a tremendous amount of message traffic and have sending histories for millions of IP addresses. Don't forget to ask each vendor how many reputations it has.

So how does a reputation system actually help your organization? Its data serves as another ingredient in the spam-detection cocktail that your company uses to help determine which messages are unwanted. Adding a measure of sender intent will definitely help make the cocktail more effective. Spam-detection cocktails use hundreds of attributes, scored and optimized to determine whether a message is spam. No one attribute is fool-proof, so in general the more data you have, the more optimized your cocktail will be. It's not that reputation data will help catch spam that no other technique would catch, but another "juror" weighing in with a guilty verdict increases the confidence level of the spam decision.

For more information:

See how image spam is clogging up networks and tricking the spam filters. Contributor Sue Hildreth reports.

Is the CAN-SPAM Act a help or a hindrance? Security expert Joel Dubin examines the effectiveness of the regulation.

Which email attacks are coming next? In this Messaging Security School lesson, Tom Bowers explains what kinds of malicious email code we can expect in the future.
The best way to use reputation is for connection management. Connection management is an additional layer of protection positioned at the perimeter, acting as a coarse filter on incoming message traffic. By checking inbound messages against data from a reputation database and discarding traffic from obviously "disreputable" senders, an organization can often block at least 50% of spam before it ever enters the network. That takes a lot of traffic away from the gateway, making it possible to do a full analysis on every message.

So what's the catch? First, as with every other spam detection technique, reputation systems are an inexact science. And every email that gets flagged at the perimeter is essentially getting the death penalty. Most vendors' systems place borderline messages in quarantine so false positives can be retrieved. But that's why most organizations set their connection management settings conservatively, so messages from only the most egregiously bad senders will be discarded.

Now spammers are not stupid, so when they realized that these reputation systems were affecting deliverability, they started looking for other ways to obscure sender identities. Since IP addresses can't be spoofed, they did the next best thing: they recruited an army of anonymous zombies to do their dirty work for them.

Zombies are actually the fatal flaw within reputation systems. Some reputation systems assume that unknown senders (which are most likely zombies) are good, and others figure they are bad. Neither method is ideal; assuming unknown senders are bad can result in more false positives, and assuming they are good inhibits the effectiveness of the connection management function. Personally, I favor systems that take a "guilty until proven innocent" approach; it's pretty clear that a great majority of the email senders out there have bad intentions. But that approach may not work for everyone. Fortunately for end users, reputation is only one piece of the spam-detection puzzle.

In today's enterprise messaging environment, there is too much spam traffic to scrutinize every message. Reputation systems used to discard spam before a message passes through the perimeter, alleviating pressure from the gateway. While they aren't the answer to all of an organization's spam woes, when used in conjunction with other technologies, reputation systems can be a valuable addition to a corporate anti-spam strategy.

About the author
Mike Rothman is president and principal analyst of Security Incite, an industry analyst firm in Atlanta, and the author of The Pragmatic CSO: 12 Steps to Being a Security Master. Rothman is also SearchSecurity.com's expert-in-residence on information security management. Get more information about the Pragmatic CSO at http://www.pragmaticcso.com, read his blog at http://blog.securityincite.com, or reach him via e-mail at mike.rothman (at) securityincite (dot) com.

This was first published in April 2007

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.