Keys to a successful network-based malware detection deployment

Network-based malware detection is an attractive alternative to traditional AV, but deployment challenges loom large. Expert Michael Cobb advises.

The exponential growth in malware variants has made endpoint security a seemingly impossible challenge. As the...

number and types of endpoints have also increased significantly, so has the importance of deploying perimeter network security technology that can scale to protect all end users, servers and traffic while coping with the growth of traffic and malware and other cyberthreats.

The extra level of analysis NBMD products offer can uncover hard-to-detect custom and polymorphic malware, often used in APT-style attacks.

Network-based malware detection (NBMD) is emerging as an alternative to signature-based endpoint antimalware software. These products are "always on" and are not susceptible to the techniques modern malware uses to defeat and bypass client-based security. However, deployments are often a challenge: NBMD needs to be inline to have the greatest effect, but without careful tuning, it can be too aggressive and disrupt mission-critical applications and business processes.

In this tip, we'll look at how to deploy inline network-based malware detection successfully, including best practices for managing and tuning the system to prevent disruptions to the application infrastructure.

NBMD advantages

Traditional antimalware detection is based on vendor-written signatures: A vendor will isolate and examine a piece of malware discovered in the wild, write a signature that tells the antimalware product what to look for, then deploy the signature to the customers of its antimalware product.

In contrast, NBMD catches known and unknown malware by actually executing any potentially suspicious file in a sandboxed environment to determine if its behavior is suspicious or malicious. The extra level of analysis NBMD products offer can uncover hard-to-detect custom and polymorphic malware, often used in advanced persistent threat, or APT-style attacks.

While a device that resides on the perimeter has low latency -- it doesn't need to send files off for analysis -- checking all traffic and unknown files on a busy network is a real challenge, even when ingress and egress points have been kept to a minimum. The latest NBMD products have responded to this issue by moving some or all of the analysis from on-box to in-cloud in order to improve cost, scalability and accuracy.

A big advantage for cloud-based NBMD services is that customers benefit from the analysis of large quantities of malware across many customers. By acting as a central repository for all file hashes, indicators and testing, the window of exposure to new malware is reduced because there is no need to distribute updated results to all on-premises devices. However, the way in which NBMD is deployed within the network has a big effect on both its effectiveness and popularity with end users.

Deploying network-based malware detection successfully

To make the most of an NBMD product, it needs to be deployed inline; like other security appliances that are deployed inline, such as firewalls and intrusion prevention systems, an NBMD product can trap and stop malware before it enters the network. An out-of-band or port mirroring deployment model means it acts more like a typical monitor, inspecting traffic and sending alerts when malware is spotted entering the network. Such a deployment doesn't scale well on-box or in-cloud, because administrators can be deluged with alerts, all of which need to be investigated and resolved quickly enough to contain any damage. Deploying NBMD inline gives enterprises more flexibility to alert and/or block.

Although blocking malware before it enters the network is a positive, automatically blocking every suspicious file from entering the network has its downsides too. False positives can potentially break critical applications and disrupt user workflow. The only way to smoothly transition to inline detection, and from alerting to blocking, is to spend time slowly tightening rules to eradicate the problems caused by false positives. Organizations should be very skeptical of vendor claims about self-learning systems; sadly, there's no shortcut around the painstaking task of defining policies and fine-tuning them over time until they work.

Initially, organizations can set the NBMD device to block only files known to be malicious, while sending alerts for any files where there is an element of uncertainty, ensuring that enough resources are available to handle the additional workload this will generate. Once it's clear certain file types don't break any processes or applications, they can be removed from alerting requirements and be blocked outright. During this period, check the logs of critical applications regularly to catch error messages, which may provide signs of key files being blocked or delayed. Warn the support desk that users may experience delays or disruptions to usual workflows and that they should get feedback on user problems, as this process will help with rule definitions.

NBMD can also be used to identify a variety of other enterprise threats, including egress network traffic that may be malicious -- for instance, patterns typical of the communication between a compromised device and an attacker's command and control center. By allowing only certain applications to send data out of the network based on indicators such as protocols, destination, time, file type and packet contents, enterprises can prevent compromised devices from sending data out of the network, and thus possibly prevent a breach.

Even with a well-tuned NBMD product in place, however, some traditional antimalware protection on endpoints is still necessary to ensure protection, regardless of whether the device is on the corporate network.

Moving forward with NBMD

In the race to supplement and eventually replace traditional antimalware programs, network-based malware detection products represent an attractive possibility for many enterprises struggling to deal with advanced threats. Though there are many hurdles that need to be jumped during an NBMD deployment, organizations that show enough persistence and commitment to tuning these devices can reap the plentiful rewards.

From the editors: More on Sandboxing

Sandboxing plays a crucial role in network-based malware detection and other advanced malware detection products. Resident network security expert Brad Casey detailed sandboxing extensively in a technical tip, including the positives to taking such an approach to malware detection and the limitations of the technology

About the author:
Michael Cobb, CISSP-ISSAP, is a renowned security author with more than 15 years of experience in the IT industry and another 16 years of experience in finance. He is founder and managing director of Cobweb Applications Ltd., a consultancy that helps companies secure their networks and websites and achieve ISO 27001 certification. He co-authored the book IIS Security and has written numerous technical articles for leading IT publications. Michael is also a Microsoft Certified Database Administrator and a Microsoft Certified Professional.

This was last published in October 2013

Dig Deeper on Real-time network monitoring and forensics