Advanced threat-detection products emerge: Benefits and challenges

Traditional security tools are no longer sufficient for defending against new breeds of attacks, forcing advanced threat-detection products to emerge.

Today's malware uses ingenious techniques to evade detection by traditional signature-based antimalware. Intrusion prevention systems, Web filtering and antivirus products are simply no longer sufficient for defending against a new breed of attacker that combines sophisticated malware with persistent remote access, with the aim of stealing sensitive corporate data over an extended period of time.

New threat-detection tools that offer advanced malware detection through sandboxing technology aim to counter this problem. These products are offered by various companies, including FireEye Inc, Damballa Inc, Palo Alto Networks, NetWitness and others. All of these systems promise near-complete protection from the malware threat. In this tip, we'll discuss the emerging techniques employed by today's advanced malware and threat-detection products, focusing on the advantages they offer and the challenges that they have yet to solve.

Threat-detection technique: Sandboxing technology

The primary technique employed by a variety of advanced malware-detection products is known as sandboxing. With sandboxing, a potential malware threat is identified using various techniques. Network traffic analysis is used to discover potential threats on the network. Patterns of behavior are analyzed, and suspicious files are sent to the sandbox. The files are then examined in an environment of virtual machines that analyze behavior in a suite of different operating systems and software versions. All changes made by the files are recorded, and a report is presented which shows all areas of the operating system and software that were changed. Based on this report, the files can be flagged as malware.

The best aspect of this approach is that, no matter the techniques used for hiding the payload of the malware, it still needs to affect the operating system in some way, and the sandboxing software will detect this. This two-stage process -- first detecting the threat, then passing it to the sandbox -- significantly reduces false positives and false negatives.

Files are also analyzed at the point of entry into the network -- for example, when they are downloaded from a website. The Web traffic is reassembled by the product, and anomalies in the code are detected and assigned priority ratings. At a certain threshold, the suspicious traffic is passed to the sandbox. Threats already on the network are minimized by blocking data exfiltration attempts based on analysis of network traffic. The malware can be prevented from making "callbacks," where the initial infection is used to download further malware. Because sandboxing is not based on signatures, it can detect brand-new malware. Information about the malware discovered then is usually shared among all the devices, which enables quicker detection time for the threat.

Threat-detection product selection process

These tools are not cheap and careful consideration should be taken to ensure that the threat-detection system you choose is right for your organization. It is definitely worth investing the time in the free trials that some vendors offer, in order to see if the system is valuable to your organization. It is important to note that once a product is chosen, it needs to be rolled out to all offices. Remote branch locations are often the starting point of attacks, and these locations need to be treated the same as any other part of the organization.

It is always important to remember that none of these products is a silver bullet. These systems cannot analyze SSL encrypted traffic, and for the most part, they are only able to analyze threats against Windows environments. They are also unable to detect malware already installed on employees' personal devices. However, the analysis of network traffic to prevent data exfiltration is a great feature to help counter some of these weaknesses.

Defense in depth is the key to preventing malware and advanced persistent threats from penetrating your network and stealing secret and confidential data. These new technologies should be seen as one layer of the defense. They should be combined with an excellent team of incident-response specialists and with frequent penetration tests to simulate real-world attacks. Highly sophisticated adversaries will frequently attack you despite the defense these products offer. As these types of products become more popular, determined attackers will attempt to develop techniques to fool the software. It is one step in an arms race that will require organizations to invest in multiple layers of defense to keep their assets and sensitive data safe.

About the author:
Robert Shapland is a network and application security expert. He has more than five years of experience as a penetration tester, and is a GWAPT-certified Web application penetration tester. He can be contacted via Twitter @rdshapland.

Next Steps

Threat-detection and management: An evolution

Network threat detection: Moving past signatures

Improving breach detection in the enterprise

This was last published in August 2013

Dig Deeper on Hacker tools and techniques: Underground hacking sites