IDS: Signature versus anomaly detection

Learn the strengths and weaknesses of signature and anomaly detection, and how the two detection methods complement each other.


What you will learn from this tip: The strengths and weaknesses of signature and anomaly detection, and how the two detection methods complement each other.

A key decision during the intrusion-detection buying process used to be whether to go with a signature or anomaly

detection engine. Where intrusion-detection systems (IDS) initially employed either detection method, vendors are becoming aware of the benefits of each and are incorporating both in their products. Understanding the strengths and weaknesses of both signature and anomaly detection reveals how they complement each another.

Signature detection

Signature detection involves searching network traffic for a series of bytes or packet sequences known to be malicious. A key advantage of this detection method is that signatures are easy to develop and understand if you know what network behavior you're trying to identify. For example, you might use a signature that looks for particular strings within an exploit payload to detect attacks that are attempting to exploit a particular buffer-overflow vulnerability. The events generated by a signature-based IDS can communicate what caused the alert. Also, pattern matching can be performed very quickly on modern systems so the amount of power needed to perform these checks is minimal for a confined rule set. For instance, if the systems you are protecting only communicate via DNS, ICMP and SMTP, all other signatures can be removed.

Signature engines also have their disadvantages. Because they only detect known attacks, a signature must be created for every attack, and novel attacks cannot be detected. Signature engines are also prone to false positives since they are commonly based on regular expressions and string matching. Both of these mechanisms merely look for strings within packets transmitting over the wire.

While signatures work well against attacks with a fixed behavioral pattern, they do not work well against the multitude of attack patterns created by a human or a worm with self-modifying behavioral characteristics. Detection is further complicated by advancing exploit technology that permits malicious users to conceal their attacks behind nop generators, payload encoders and encrypted data channels. The overall ability of a signature engine to scale against these changes is hamstrung by the fact that a new signature must be created for each variation, and as the rule set grows, the engine performance inevitably slows down. This is the very reason that most intrusion-detection appliances reside hardware that runs from two to as many as eight processors with multiple Gigabit network cards.

Essentially, the signature-based IDS boils down to an arms race between attackers and IDS signature developers, where the delta is the speed at which new signatures can be written and applied to the IDS engine.

Anomaly detection

The anomaly detection technique centers on the concept of a baseline for network behavior. This baseline is a description of accepted network behavior, which is learned or specified by the network administrators, or both. Events in an anomaly detection engine are caused by any behaviors that fall outside the predefined or accepted model of behavior.

An integral part of baselining network behavior is the engine's ability to dissect protocols at all layers. For every protocol that is being monitored, the engine must possess the ability to decode and process the protocol in order to understand its goal and the payload. This protocol "dissection" is initially computationally expensive, but it allows the engine to scale as the rule set grows and alert with fewer false positives when variances from the accepted behaviors are detected.

A disadvantage of anomaly-detection engines is the difficultly of defining rules. Each protocol being analyzed must be defined, implemented and tested for accuracy. The rule development process is also compounded by differences in vendor implementations of the various protocols. Custom protocols traversing the network cannot be analyzed without great effort. Moreover, detailed knowledge of normal network behavior must be constructed and transferred into the engine memory for detection to occur correctly. On the other hand, once a protocol has been built and a behavior defined, the engine can scale more quickly and easily than the signature-based model because a new signature does not have to be created for every attack and potential variant.

Another pitfall of anomaly detection is that malicious activity that falls within normal usage patterns is not detected. An activity such as directory traversal on a targeted vulnerable server, which complies with network protocol, easily goes unnoticed since it does not trigger any out-of-protocol, payload or bandwidth limitation flags.

However, anomaly detection has an advantage over signature-based engines in that a new attack for which a signature does not exist can be detected if it falls out of the normal traffic patterns. The best example of this is how such systems detect new automated worms. When a new system is infected with a worm it usually starts scanning for other vulnerable systems at an accelerated or abnormal rate flooding the network with malicious traffic, thus triggering a TCP connection or bandwidth abnormality rule.

You can see how the strengths of one detection method benefit the weaknesses of another and vice versa. Choosing a detection method is no longer an either/or proposition when buying an IDS.


MORE INFORMATION:

James C. Foster is the Deputy Director for Global Security Solution Development for Computer Sciences Corporation and the lead author for the new Syngress Application Security Series; Mr. Foster can be contacted at jamescfoster@gmail.com.


This was first published in May 2005

Dig deeper on Network Behavior Anomaly Detection (NBAD)

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

SearchCloudSecurity

SearchNetworking

SearchCIO

SearchConsumerization

SearchEnterpriseDesktop

SearchCloudComputing

ComputerWeekly

Close