Information Security

Defending the digital infrastructure


Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

A Safe Bet?

Network anomaly detection is the newest player at the security table.

Network anomaly detection is the newest player at the security table.

Case Study: Boyd Gaming Rolls a Winner With NADS

Few understand risk better than a casino operator. That's why Les Leonard, director of IT for Boyd Gaming, doesn't take any chances when it comes to securing his IT infrastructure. In addition to conventional layered security defenses, Leonard relies on anomaly detection to protect the $1.7 billion gaming company from zero-day attacks and insider abuse.

About a year ago, Leonard deployed Lancope's StealthWatch and the StealthWatch management console — and he soon realized that he wasn't aware of everything happening on Boyd's network. Leonard says StealthWatch helps him spot "things that are occurring that shouldn't be occurring." Recently, StealthWatch spotted and blocked a South Korean lottery company trying to use one of Boyd's Web servers as a spamming device. "That's the kind of thing it's great at," he says.

Boyd has been running StealthWatch for more than a year now, and Leonard says it has helped block the MS Blaster worm and spot unauthorized P2P applications that posed security and legal risks.

Lane Timmons, systems analyst at the Department of Health Sciences at Texas Tech University, is no stranger to anomaly detection systems either. About three years ago, he deployed Q1 Labs' QRadar. Since then, QRadar has spotted new malware attacks and anomalous employee behavior. "The zero-day thing is what I really want it for; that's when it really comes in handy," says Timmons. Recently, QRadar spotted a modified version of the SDbot worm that infected about 30 of the department's 7,500 desktops, before McAfee — Texas Tech's AV provider — had released a signature.

"It was doing anonymous enumeration on the domain controllers and locking out our accounts. We got serious about it right quick," says Timmons. "We got it before McAfee even knew about it." QRadar also proved helpful when French hackers broke into one of the university's systems to illegally swap movies. While Timmons' IDS spotted the activity, it didn't provide much in the way of details. "All the IDS did was fire off two vague alerts," says Timmons. He used QRadar to locate and shut out all of the outside systems that had used FTP to connect to the hacked system.

While anomaly detection systems have spotted new attacks, both Leonard and Timmons say the systems also help keep close tabs on employees who may be violating security policy. "You have to look at employees, too. Maybe you have employees who start doing things that they shouldn't be doing. It's good with the insider threat, which is probably the biggest threat you have," Leonard says.

"We had PC support techs not doing their job. We set up QRadar to watch their subnet and see what Web sites they were going to. The bosses like to be able to see that stuff," says Timmons.

And that's the key driver for anomaly detection systems: Security managers don't know what they don't know about the activities on their network. "It helps us in ways that we didn't anticipate, because we didn't realize all of the stuff that was going on out there," says Leonard.

George V. Hulme is a freelance security writer based in Minnesota. Send your thoughts on this article to

To some degree, network anomaly detection systems (NADS) are like Rodney Dangerfield: They get no respect in a security monitoring game that has numerous variables.

Network anomaly detection isn't exactly intrusion detection or prevention. It's not a firewall. It's not directly applicable to compliance. And, while it has the ability to automatically enforce policy, enterprises aren't really using that capability.

Vendor marketing materials promise that NADS will deliver automated profiling and tuning, zero-day exploit detection, and the ability to address events missed by traditional IDS/IPS. While enterprises aren't necessarily finding the total fulfillment of that promise, these systems have shown their value as a part of comprehensive enterprise monitoring.

In general, NADS are designed to analyze network traffic with data gathered from protocols like Cisco Systems's NetFlow, Juniper's cFlow or sources that support the sFlow standard. Data is correlated directly from packet analysis; and the systems use a combination of anomaly and signature detection to alert network and security managers of suspicious activity, and present a picture of network activity for analysis and response.

NAD: The Real Deal

Network anomaly detection has been around for a few years, and the concept of anomaly detection in network defense even longer. The following debunks a few myths surrounding NAD:


Automated profiling/modeling/training eliminates tuning.

Automated response capabilities to enable containment, like

pushing ACLs to network infrastructure, can protect against threats.

NAD catches hard-to-detect insider misuse that cannot be addressed by firewalls and IPS.

NAD detects zero-day exploits.

NAD is the last line of defense.


It doesn't eliminate it; it changes the nature of the tuning from which signatures should be enabled to which anomalies you choose to highlight and address.

Automated response features are present, but — as was the case with early adopters of IPS products — most operators are reluctant to turn on automated response because of the high potential for false positives. While NAD produces interesting intelligence, the collected data is better suited for investigative rather than automated responses.

Technically, yes, but it's subject to the limitations of determining which anomalies are really important. NAD doesn't substantially affect the difficulty of detecting and addressing malicious insiders.

NAD detects infections and helps organizations contain them before they spread, which could be argued as the only really effective way to deal with zero-day attacks. There's no magic that understands zero-day exploits any better than any other technology.

This may be true. As a decision-support system, NAD helps organizations address the impacts of various attacks and behaviors on their network.

Not Really an IPS
To understand the value and limitations of NAD technology, it helps to understand the differences between it and a traditional IDS/IPS. NADS are more complementary than competitive solutions to the signature-based technologies. While the systems have the ability to trigger automated responses and/or blocking like IDS/IPS, they're used most frequently to create a presentation of the network traffic that allows a security manager with good contextual knowledge of his network to identify otherwise elusive behavior. They can be thought of as a last line of defense when preventive tools like firewalls and IPSes fail to stop real-time exploitation of vulnerabilities. And, they can be used to detect new applications, behavior and devices for investigation.

The most significant difference between NADS and IPS/IDS is the class of attacks each addresses. An IPS primarily detects (and possibly blocks) attacks that can be predefined. The size and comprehensiveness of the attacks that can be blocked is dependent on many factors, such as placement of the sensors and quality of the signatures.

Despite vendor claims to the contrary, NAD is primarily an investigative technology. While it has the potential to detect zero-day and other stealthy attacks, confidence in its results remains a problem in enabling automated response mechanisms. This isn't unlike the early versions of IDS/IPS products, which weren't reliable enough to enable automated responses. In this light, NAD is best used to detect, investigate and manually address suspected incidents and problems.

Herein lies its advantage and differentiation from IDS/IPS: NADS may not be able to automatically detect and block with the confidence of an IPS signature, but neither can an IDS/IPS help an organization if the enabled signature set misses something. This is where NAD shines.

One company reported installing a NAD product after Nimda hit its network, penetrated 70 percent of its servers and took four days to bring under control. With NAD firmly in place, the company was then hit by the MyTob virus. The system discovered MyTob on five PCs within a few minutes of the first infection, and sysadmins manually blocked ports to contain the virus. This is a typical success story with NADS, where they provide early warning and investigation to isolate the problem, but aren't used for automated response.

In addition to detecting ongoing attacks, NADS can work with policy compliance or vulnerability management applications to monitor and prevent incidents. For instance, NADS can detect machine-to-machine connections, unauthorized applications and processes, protocol and port usage, misconfigured systems, network traffic loads and bandwidth consumption.

Technology Forecast

Many organizations will find network anomaly detection systems (NADS) useful as complementary controls to firewalls and IDS/IPS implementations. However, convergence in both the technology and its vendors continues. Eventually, NAD technology will become a feature set of comprehensive network security suites.

NADS providers have begun adding the ability to accept other data sources, such as firewalls, creating an overlap with security event management (SEM) products. SEMs, meanwhile, are adding statistical analysis and are starting to overlap with NADS. Finally, IDS/IPS vendors such as Sourcefire and Enterasys Networks (through a partnership with Q1 Labs) are adding network flow analysis to their products' capabilities.

The convergence of anomaly detection with conventional security technologies is best illustrated by several emerging and established vendors. ConSentry Networks is blurring the line between signature-based IPS and NADS with a hybrid inline system that analyzes and controls traffic. Startup Intrusic operates out of band, but directly analyzes packets for high-value patterns of interest like reverse tunneling.

While these capabilities may not survive as standalone systems, they have proven their worth in assisting organizations with monitoring requirements. A decade of learning has taught us that monitoring is fundamentally flawed and that separating good from bad behavior is significant. Network anomaly detection systems offer a different view of network activity, which focuses on abnormal behaviors without necessarily designating them good or bad.


NADS Is Limited
Security incident detection typically falls into two categories: signature- and anomaly-based. These terms are so overused that they have lost all common understanding. There are several technology implementations and techniques that fall within these two categories, including mechanisms — protocol analysis, expert systems and artificial-intelligence techniques (neural nets, Markov chaining, etc.) — but there are some fundamental properties of each.

Signature mechanisms detect patterns of predefined sequences in data — event A followed by event B followed by event C. If implemented and tuned properly, signature mechanisms have relatively few false positives. Of course, they will only detect attacks and incidents for which they have signatures, so the must be kept up to date.

Anomaly mechanisms model good behavior and look for deviations from the baseline. For example, an observation of network traffic shows that event B has always followed event A, so it would be considered anomalous if A followed B. This can be rife with false positives because it is difficult to model all possible good behavior. It requires stable systems that maintain a consistent model of what constitutes good behavior; the trick is figuring out which events should be measured and which are significant.

True anomaly detection means that observable (normal) behavior is modeled, and departures from that behavior are flagged. For example, certain identifiable protocols (TCP, IP, HTTP, ARP) are present at certain times (hours in the day, day of week) and at certain throughputs (100, 200, 500 Mbps). Once a baseline of behavior has been established, an anomaly would be anything that departs from this pattern — like the FTP protocol appearing or traffic volume changing during the day. These anomalies ostensibly indicate occurrences on the network that may be of interest to the security team.

NADS use network flow data to track predefined measures for establishing a baseline of normal behavior and anomaly analysis. Most of these systems use the concept of an index to indicate confidence in both how anomalous something is and how sure the system is that the anomalous activity is relevant and security-related. NAD's implied advantages include reduced tuning complexity, because it's not prone to human error, and zero-day detection, since it's not reliant on predefined signatures. However, the value of these advantages is significantly impacted by the volatility of enterprise networks. NADS suffer from high false-positive rates when "good" behavior can't be effectively modeled. Factors that affect network behavior modeling include the following:

  • The number of possible behaviors and event types. Larger numbers of behaviors and event types yield more combinations of good and bad, which adds to complexity.
  • The stability and consistency of the environment. A network with poor change control (common in many large enterprises) or a highly dynamic environment may have applications, partners or devices regularly changing traffic patterns, which reduces the model's reliability.
  • The stability and consistency of network activity. A network subject to bursts of activity of varying types will make modeling both good and bad behaviors difficult.
  • The reliability of bad behaviors. When unusual behavior isn't always bad, the gray areas can weaken the modeling and increase false positives. For example: Is it always bad when a computer that has never used a particular protocol starts to use it?

Anomaly detection is really only useful in stable environments where "normal" behavior can be effectively modeled. NADS vendors will admit that their products are only useful in controlled environments, such as an LAN, and unsuitable in unregulated settings, such as at the gateway. A good example is network protocol anomaly detection (provided by most contemporary IDS, IPS and NADS products), where the roughly 300 RFCs provide a model of normal behavior in which departures from the model are significant and useful for detecting attack traffic. Of course, many application developers regularly violate the RFCs, further illustrating the limitations of modeling even within well-defined rules.

While an argument could be made that tracking a sufficiently specific set of criteria could yield more useful results, experience has shown that more narrow criteria increase the number of events that must be investigated and the number that aren't truly of interest. For example, it may be suspicious if the application development LAN FTPs from the finance LAN, but this implies that new FTP connections, while clearly anomalies, are suspicious, and you will get flagged every time an FTP connection is made that exceeds defined thresholds. Some of those connections will clearly be interesting, suspicious and useful to know about; however, the majority will be acceptable business activity.

NADS users report that narrowly defined activities, like a new service starting, a new protocol appearing, or a new port opening, are interesting in theory, but that granular anomaly detection isn't useful in a large enterprise where those activities are common. Therefore, the usefulness of NADS in any particular environment is a function of the stability of the network, which behaviors are modeled, and the security manager's knowledge of the organization's network and the meaningfulness of the deviations. Enterprises that have had success with NADS are the ones that have done the most effective job of balancing and tuning these parameters.


The following is a representative list of pure-play network anomaly detection systems vendors and products.

Arbor Networks -

Peakflow monitors network traffic for deviations in load, providing insight into changes in network behavior.

Captus Networks -

Captus IPS 4000 series is a combined solution for network management and security, which prevents DDoS attacks, port scans, unknown worms and unsanctioned traffic.

Lancope -

StealthWatch provides enterprises with a hybrid anomaly detection and response system that bases its actions on behavior deviations and protocol analysis.

Mazu Networks -

Its two products, Profiler and Enforcer, work in concert to monitor for deviations and automatically respond to threats.

Q1 Labs -

QRadar provides security managers with a continuous analysis of network traffic flow, giving real-time analysis of traffic type and bandwidth consumption.

Narus -

Born as a network management company, Narus provides anomaly detection products and technology to carriers.

netZentry -

Its FloodGuard technology provides enterprises with non-signature-based defenses against DDoS attacks, botnets and syn floods.w

Place a Small Bet on NADS
NAD devices are powerful knowledge tools for expert network operations people with enterprise-specific contextual knowledge. These systems can help enterprises learn about the traffic and behavior of their network. Even though they can catch detailed events, such as a new service opening up, a new protocol appearing or a new machine connecting to the network, these events are too common to have value in larger enterprises. NADS shine where obvious behaviors — like when a worm-infected machine spewing attack traffic or a DoS attack — are under way. The value these systems offer for addressing more subtle behavior is dependent upon the knowledge and experience of the operator. Under the right circumstances, NADS provide a wealth of network behavior information (protocols, ports, services, throughput, latency, etc.) that can be used to understand what's really going on in your network.

While network operations and security experts may find this cornucopia of network information empowering, it may be overwhelming to a person without the context and tribal knowledge of the enterprise-specific network infrastructure.

Article 1 of 15

Dig Deeper on Network intrusion detection and prevention (IDS-IPS)

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

Get More Information Security

Access to all of our back issues View All