Achieving network security with tomorrow's antivirus tools

by Joel Snyder

Intrusion Defense School Move to the next lesson: Unified Threat Management Return to Intrusion Defense School

Viruses, worms and Trojan horses represent a constant threat to enterprise networks. As a class, malware has become not only one of the greatest risks to network security, but also one of the areas best covered by existing defenses. While the debates over the usefulness and appropriateness of technologies such as intrusion prevention and content filtering continue, no one seems to disagree with the premise that true network security requires a strong antivirus strategy.

Because the threat level and risks of malware are high, enterprises and software vendors have sought additional tools beyond traditional virus scanners to mitigate risks.

The most promising new anti-malware technologies are reputation-based systems, both for spam and viruses. As a network perimeter tool, reputation-based defense systems are quickly becoming the norm in a wide variety of products, from the IP layer up to the application layer. However, network security managers should look at a variety of alternative antivirus techniques, ranging from the low-tech filtering of attachments to more innovative in-line and heuristic-based antivirus systems.

Going inline
Antivirus tools are some of the oldest technologies we have for network perimeter intrusion defense. At the edge of the network, classic antivirus technology is usually incorporated into dedicated email security appliances designed to scan incoming (and sometimes outgoing) email for malware. That's an active and full market, with literally over a hundred products available to act as email security devices.

When an unchecked malware infection occurs, the risk to a network can be enormous. Although we haven't seen any disastrous viruses lately, many security managers are still very aware of the havoc that worms such as Code Red, SoBig and Sasser caused.

As a further precaution, some enterprises also perform in-line antivirus scanning. In-line antivirus scanners, typically incorporated into firewalls, look at not just incoming and outgoing SMTP traffic, but also other mail protocols (POP and IMAP), Web traffic (HTTP) and often file transfers using FTP. While in-line antivirus scanners are not as flexible or as reliable as standalone antivirus, they can catch a large percentage of virus traffic and can be a valuable adjunct to both desktop and server-based antivirus deployments.

The danger with in-line antivirus is that it's much more difficult to perform antiviral checking on packets flowing across a wire than it is on a file sitting on the end-user's disk. For example, if an end-user uses an SSL connection to read their Yahoo email account, the packets are encrypted -- and the inline virus scanner can't see them. Even if they don't use an encrypted connection, how Webmail services deliver attachments varies from service to service, and an antivirus scanner must be able to reconstruct the file based on what the user sees. That can mean special application-specific intelligence for popular Webmail services and a risk that tomorrow's new Webmail service or gateway won't be properly scanned.

In-line scanners are also hampered because they can only scan traffic they understand, in protocols they expect. While you can often assume that Web traffic will run over TCP port 80 and SMTP over TCP port 25, those assumptions are just unreliable best guesses, especially if the user or software is trying a more crafty approach. An in-line antivirus scanner could, in theory, be looking at all traffic over all ports. However, performance issues related to this approach make it untenable for all but the slowest and smallest of networks. Most Unified Threat Management (UTM)-based antivirus products won't even offer this option because of the performance costs. A simple check box that says "scan all ports" and takes the firewall down to its knees is a dangerous option.

Most in-line network-based intrusion prevention system (IPS) products include some antivirus capabilities. No IPS advertises itself as a full antivirus solution, but some of the most egregious malware can be detected by an IPS.

IPSes are much better at catching malware after it has already infected your network, during propagation. Almost any malware that actively propagates can be detected by its behavior. For example, a malware-infected PC might suddenly start making random connections to mail servers (rather than using the corporate mail server). Or it might try to connect to hundreds or thousands of file servers across the corporate network. This type of propagation can be easily detected by most IPS technologies, and you'll find that IPSes have the capability to block further malware propagation, based on behavior anomalies alone.

If you are looking into in-line anti-malware technology, here are some guidelines that will help you choose and configure the best solution:

1. Because in-line anti-malware does not have total control over the client/server communication that it's intercepting, the notification that a policy violation has occurred can be clumsy. For each of the paths that you plan to intercept, be sure that you understand exactly what an end-user is going to see and how the Web or mail client will respond when inline anti-malware has triggered. This knowledge will be an important part of your security training for end users, as well as for the help desk, in solving "strange behaviors" with client systems.

2. Inline anti-malware can check both incoming and outgoing traffic. While it's great to prevent infections, it's almost as good to know who is infected. Make sure you enable outgoing traffic checking, and create an alerting system so that you know who has become infected. Actively responding to those alerts will help to contain infections and assist end users in understanding what action they might have taken that caused the problem.

The problem with antivirus scanners in email is that there is always a lag between the time a new virus is created and the time when an antivirus scanner has a signature that can catch the virus. Of course, many viral variants and polymorphic viruses can be caught by existing signatures, but there will always be new examples that require signatures or, worse, changes in the scanner to catch them.

Finding malware before a definitive signature is available is one of the goals of any good security program, and network security managers have turned to a variety of antivirus techniques to help them reach that goal.

One low-tech but surprisingly effective technique has been to simply block many types of attachments (technically MIME body parts) in email messages. The logic is pretty simple: most people in organizations are sharing a small subset of the world of file types, typically Microsoft Office documents, text files and Adobe PDF formatted files. Rather than allow anyone in the organization to get anything from the outside world, network managers have set filtering policies to either only allow specific types of attachments or to block some of the types that are well-known as infection vectors (such as executable files or batch files).

It's a somewhat draconian policy, but it also turns out to be an effective one for protecting against many of the existing viral threats, especially in the enterprise environment. Personally, I don't care for this strategy. I think that we buy virus scanners and other anti-malware tools to do this for us and picking up a blunt stick to bludgeon email into submission has higher costs than benefits. However, every environment is different and there are network managers who have evaluated their environment and decided that this is the best strategy. If you choose (or already have chosen) a policy like this, four guidelines will help you arrive at the best solution:

1. Email attachments typically have three labels on them: a filename extension, a MIME type and a real file fingerprint, determined by looking at the file and checking the actual file type. Your filtering solution should act on all three of these, not just one. If it turns out that an executable simply has to be renamed to .XLS to get through your filter, then people will discover that in a few minutes and you'll have reduced total effectiveness.

2. If you're going to have a policy like this, your best strategy is to have a deny-by-default approach with certain file types allowed. Otherwise, you'll be forever updating the list of "bad" attachment types as new viruses figure out a way to get through common defenses. Once you come up with a reasonable set of file types for your user community, advertise what is and what is not permitted so that this is not a surprise to end users. Don't forget that people are increasingly using archiving tools, such as ZIP, to package up multiple files. You will need to allow at least one of these (but probably not every variation).

3. Understand that end users will find ways to work around this policy. The availability of tools like YouSendIt.com (a Web site that allows anyone to upload documents transparently for later download) means that you won't be able to keep every file type out. Since users will always find valid exceptions to this kind of control, you need to both educate them on why you're doing this and how they can safely work around your controls. You also need to be able to make exceptions to the policy so that people can get their work done. If the marketing department is paying someone to generate a new logo in Adobe Illustrator, you need to have a quick way to let them conveniently move that file type around.

4. While most malware should be eradicated and the carrier (such as an email message) deleted without notice, this isn't true of deleted attachments. Anytime you take anything out of an email message, you should put something back in to indicate the action you've taken. Obviously, this means that any attachment filtering should happen after normal virus scanning.

Heuristic-based antivirus is a fairly old idea, but one that has not gained a strong following. The idea is that viruses can be identified not just by their signature, but by heuristics that look for unusual patterns in certain types of files for example, or by executing the virus in a virtual machine "jail" to examine its behavior. Many antivirus vendors include some heuristic technologies in their scanning engines, but no one has gone out on a limb and suggested replacing signature-based antivirus with heuristic-based systems. Heuristic-based antivirus has more variable and unpredictable false-positive and false-negative rates than normal antivirus software, because the rules used to identify malware are fuzzier and approximate than with a signature-based product.

From a network manager's point of view, the appearance of a heuristic-based technology is a positive sign that an antivirus vendor is looking for more than just signatures in searching for viruses. However, it's unlikely that heuristic-based antivirus will ever push aside signature-based technologies.

Reputation-based antivirus uses the context of the virus to help decide whether an attachment or connection is likely to have undesirable consequences. Like heuristic-based antivirus, reputation-based systems are an adjunct to virus scanners. Vendors are promoting them for "near zero-day" protection, because they can be used to identify suspect email before an antivirus signature is prepared.

Reputation-based services vary widely in their actual implementation, but have some common elements, including a large centralized reputation database and a voracious appetite for transaction data about the behavior of systems across the Internet. For example, a reputation-based service might collect information about spam and virus senders from its customers and other spam traps. When a system or network is identified as being a source for malware or spam, future messages can be held up or specially tagged because of the poor reputation of the sender.

This can be taken a step further: if you actually accept the suspect messages from suspect senders, you can analyze what they're sending and use that to characterize likely spam or malware from other senders. Reputation-based services use this technique and others, to help identify viruses before the antivirus signatures can.

If new techniques like these interest you, here are some guidelines that will help you choose and configure the best solution:

1. There is no magic in this business. When looking at new technologies, a vendor should be able to explain what they are doing and how this improves on standard signature-based antivirus. Any pitch with excessive hocus-pocus or claims of revolutionary advances is likely accompanied by an ineffective product that won't stay the distance.

2. Reputation-based technology is one of the most effective ways to knock the top third of "bad traffic" off your network. As long as spammers and malware writers use zombie PCs, this kind of technology will be very valuable. Anyone looking to increase system performance and gain an edge on the malware of tomorrow should be looking carefully at reputation-based services.

Return to Intrusion Defense School

BROWSE BY TAG Network Intrusion Detection (IDS),   Network Intrusion Detection and Analysis,   Enterprise Network Security,   Network Intrusion Prevention (IPS),   Denial of Service (DoS) Attack Prevention,   Intrusion Defense School,   Network content security: Antivirus and antispyware tools,   VIEW ALL TAGS

Digg This!     StumbleUpon     Del.icio.us

<< PREVIOUS | NEXT >>

VIEW ALL IN THIS CATEGORY

RELATED CONTENT Network Intrusion Detection (IDS) Preventing SQL injection attacks: A network admin's perspective Lifecycle of a network security vulnerability Best Intrusion Prevention and Detection Products Rogue AP containment methods SIMs tools and tactics for business intelligence IPS and IDS deployment strategies Know when you need IDS, IPS or both Trend Micro to acquire Third Brigade for virtualization, cloud security New product aims to control rogue applications that avoid firewalls How to perform a network forensic analysis and investigation Network Intrusion Detection (IDS) Research Network Intrusion Prevention (IPS) Aligning network security with business priorities Best Intrusion Prevention and Detection Products Port scan attack prevention best practices Lesson 4: How to use wireless IPS Lesson 1 quiz: Risky business Hacker attack techniques and tactics: Understanding hacking strategies SIMs tools and tactics for business intelligence IPS and IDS deployment strategies I'll be watching you: Wireless IPS Know when you need IDS, IPS or both Network Intrusion Prevention (IPS) Research Denial of Service (DoS) Attack Prevention VeriSign extends DDoS attack protection service Conficker authors prepping for next stage, researcher says Latest DDoS attacks extremely unsophisticated, experts say DDoS attacks hit U.S., South Korean government websites How to prevent a denial-of-service (DoS) attack I'll be watching you: Wireless IPS How to prevent DDoS attacks on websites How to prevent network denial-of-service attacks What are 'phlashing' attacks? Could someone place a rootkit on an internal network through a router? Denial of Service (DoS) Attack Prevention Research

RELATED GLOSSARY TERMS Terms from Whatis.com − the technology online dictionary computer forensics  (SearchSecurity.com) Diffie-Hellman key exchange  (SearchSecurity.com) Einstein  (SearchSecurity.com) HIDS/NIDS  (SearchSecurity.com) network behavior analysis  (SearchSecurity.com) ultrasound  (SearchSecurity.com)

RELATED RESOURCES 2020software.com, trial software downloads for accounting software, ERP software, CRM software and business software systems Search Bitpipe.com for the latest white papers and business webcasts Whatis.com, the online computer dictionary