This content is part of the Security School: Knock advanced malware out cold with network anomaly detection
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Find network anomalies and you'll ax advanced malware

Learn how advanced malware evades perimeter defenses and why tools to detect network anomalies are essential to keep your network secure.

In this two-part tip you will learn how malware can now evade traditional enterprise security systems. In part...

1, here, our expert discusses the nature of advanced malware and how it operates. In part 2, readers will learn why security tools that detect network anomalies can thwart sophisticated malware.

As most security pros are well aware, malware is code, a program or software inserted into a computer or network system to compromise a target's data, applications and operating system, or otherwise disrupt operations of the victim. Whenever there's a new data breach, account compromise or denial-of-service attack, malware is usually the weapon used.

Recently, talk about malware has started to center on a new breed -- so-called advanced malware. What makes malware advanced, you ask? There's no one answer, but the industry has come to broadly define it as malware with new and sophisticated capabilities that distinguishes if from thousands of new-yet-ordinary malware that appears daily on the Internet. These advanced capabilities include: an ability to hide from detection for long periods (through encryption, for example); an ability to target an individual or small group, often by compromising previously unknown flaws (also known as zero days); and an ability to attack a number of vulnerabilities by combining a multitude of techniques.

Today's enterprises struggle to defend themselves against advanced malware. As research shows, some organizations haven't yet realized their security focus must extend well beyond their network perimeters; those that have attempted to do this often struggle to get the funding they need for the technology and trained staff that an effective advanced malware defense requires. And even if an organization has virtually every possible defense in place, it can still be compromised, because the evasion techniques advanced malware employs are notoriously difficult to identify and stop.

Thus it's no surprise that enterprises are doing such a poor job in detecting intrusions. In several of the recent large retail network attacks in the United States, the network owners never discovered the intrusions, but rather were informed by third parties, including law enforcement and -- in one case -- a security blogger. Unfortunately, by the time a third party notices that tens of thousands of credit cards from a given retailer are being sold on the underground market, the opportunity to quickly detect and stop the intrusion has been lost.

Protection strategies

What about the use of common technical controls at the border? Why aren't they working to detect advanced malware? Let's look at two common technical security products: the firewall and intrusion detection/prevention systems.


A firewall is a default-deny control device. Similar to a router, a firewall forms a boundary between networks. As a control device, a firewall can "decide" to either allow or deny traffic through the boundary. The firewall makes its decision based upon rules that an administrator has applied. Default-deny means that in the absence of a rule that permits network traffic, the traffic is denied and not allowed to cross the boundary.

It can be difficult to know what traffic should be permitted across the boundary. A clearly written firewall security policy can help. In the absence of a firewall security policy (and in the spirit of causing the fewest problems for users) firewall rule sets can be very permissive. Sometimes firewall rule sets don't progress beyond the original default configuration the manufacturer ships with the product.

Intrusion Detection and Intrusion Prevention systems

An intrusion detection system (IDS) is a passive monitor that observes network traffic and attempts to search for malicious activity by matching traffic with a set of rules or signatures; it works much like an antivirus system. If a match is discovered, an alert is sent to an administrator and a security console. The problem is that there will be no recognition of malicious traffic, and therefore no alert, unless:

•Malicious traffic is publicly known;

•A signature has been developed for that malware; and

•Signatures for that malware have been installed.

Beyond these conditions, there is the problem of a security team not responding to alerts in a timely fashion -- or not responding to them at all.

An intrusion prevention device (IPS) is default-allow control device. If an IPS matches traffic with a signature in its database, then it can stop the traffic, much like a firewall can. Many IPS systems are installed on firewalls. 

An IPS system has exactly the same problems that an IDS has with signatures, making both the IDS and IPS security solution ineffective against advanced malware.

What's needed now

Because IDS and IPS cannot adequately protect against advanced malware, security pros must change their focus and do more than just try to detect and deny malware at the border. They must acquire and deploy tools that examine the interior of the network as well as the perimeter, tools that possess the ability to detect network anomalies.

Part 2 of this tip will look at security these types of tools, how they work and why, in an age of advanced malware, they are essential to adequately secure the enterprise network.

About the author: Peter Sullivan began his career in network operations, information security and incident response 20 years ago with the U.S. Army. For the last ten years, Sullivan has been a visiting scientist at the Software Engineering Institute, Carnegie Mellon University, where he teaches courses in risk management, information security and assurance, computer security incident response and digital forensics. He is also a partner with InfoSecure Solutions, LLC, a Massachusetts based consultancy specializing in IT risk management and incident response planning. Sullivan holds a CISSP certification and a CERT/CC Computer Security Incident Handling (CSIH) certification.

This was last published in June 2015

Dig Deeper on Malware, virus, Trojan and spyware protection and removal