Problem solve Get help with specific problems with your technologies, process and projects.

Why sandboxing technology is integral for advanced malware detection

Expert Brad Casey details how advanced malware detection products rely heavily on sandboxing technology, though it's not a cure all for enterprises.

Signature-based malware detection products are in the twilight of their useful life. The rapid evolution of malware...

for which no signature exists and the ongoing use of attack methods that don't rely on malware (i.e., stolen credentials) have greatly reduced the effectiveness of signature-based AV.

Many enterprises are likely questioning the value they receive from traditional signature-based malware detection products, with perhaps the most famous recent criticism coming from The New York Times, which, in January, revealed it had been the victim of a China-based cyberattack campaign that had gone on undetected for at least four months despite the newspaper's use of Symantec antimalware products.

To bridge this malware-detection gap, many within the security industry have proposed a paradigmatic shift from signature-based malware detection to new, more proactive detection systems. A new enterprise information security product segment, advanced malware detection products, has emerged to fill the need left by traditional antivirus products, but just what new methods are they bringing to the table that would garner enterprise interest? And can enterprises completely cast off traditional antivirus in favor of this new antimalware breed? That's what we'll discuss in this tip.

Sandboxing: A key technology in advanced malware detection

Before delving into new malware detection methods, let's first cover how traditional antivirus products work.

In the signature-based malware detection paradigm, researchers are constantly on the lookout for malicious code that does not have a detection signature associated with it. Once they find such a sample, the security researchers attempt to write signatures that allow antimalware software to block and/or mitigate the newly discovered malicious code. The primary problem with this approach is that it is painfully reactive. Newly formulated malicious code all too often goes undiscovered until it has already been successfully executed against at least one victim, hence the need for more proactive malware detection.

Amid the ongoing effort to replace signature-based detection, sandboxing technology has emerged as key to the future of proactive malware detection. Simply put, sandboxing is the practice of taking inbound network traffic and diverting it toward a separate virtual environment. Once each data packet is examined and/or executed in the virtual environment, the traffic is either forwarded to its intended destination or deleted if deemed malicious. If the malware is particularly nasty and the system administrator is unable to delete the malicious code, the administrator simply deletes the entire virtual environment and rebuilds it later.

Vendor implementations

Several vendors have emerged with unique spins on advanced malware detection; three in particular have garnered significant market traction: FireEye Inc., Damballa Inc. and Invincea Inc.

FireEye has enjoyed a rapid gain in popularity recently as it was one of the first vendors to offer sandbox technologies at the enterprise level. Its on-premises system uses the typical sandbox approach of diverting traffic, examining it and then forwarding it. As email is considered a latency-prone form of network traffic, FireEye technology seems to be more effective at combating malicious email then other types of typical Web traffic. Simply put, email will get there when it gets there, so the FireEye system can really take its time in examining each email that traverses its path.

Damballa, like FireEye, specializes in nonsignature-based malware detection. However, Damballa's approach takes the form of detecting botnet or botnet-like communications. In a typical botnet, the orchestrator is known as the botnet master. The master injects malicious code into a victim machine, which becomes the command and control (C&C) node due to its lightweight network presence along with its ability to communicate in near real time. Where Damballa technology comes into play is in the detection of typical botnet communication protocols, such as the Internet Relay Chat protocol, which is popular with botnet masters because of its lightweight network presence along with its ability to communicate in near real time. Whenever this type of protocol is detected, a red flag is raised and Damballa technology moves in for deeper inspection. If the traffic is deemed to be associated with a bot, the communication is immediately shut down.

Lastly, Invincea took the sandboxing concept and placed a fascinating spin on it. Rather than diverting all inbound traffic into a virtual operating system, Invincea simply virtualizes the Web browser, and when an end user opens a Web browser, they're actually interacting with a browser that is sitting inside the Invincea enclave. Invincea argues that it knows how the ideal browser should perform on a day-to-day basis, so when its system detects anomalies in the performance of the browser, deeper inspection is conducted. For example, if a piece of inbound traffic requires that the browser make a system call to the host operating system and Invincea deems the system call to be unusual, a forensic inspection is initiated. If the code is deemed malicious, the entire browsing environment is deleted and automatically rebuilt. This process is supposedly transparent to the end user, and if this is indeed the case, Invincea has taken a rather novel approach to detecting advanced malware.

Problem solved?

Obviously, significant strides have been made in the arena of proactive malware detection. No longer are organizations completely dependent on antimalware vendors to provide timely and accurate signatures after a major attack has already been executed.

From the editors: Choosing antimalware products

Is your organization looking for the right antimalware suite? Mike Rothman of Securosis provides an entire guide for buying antimalware products, from assessing technical considerations to questions for vendors.

However, enterprises should exercise caution before deciding to rely solely on a single advanced malware detection product. In a number of sandbox implementations, an emphasis is placed on running potentially malicious code in a separate environment prior to forwarding the packets to the intended destination. As malware authors use more sophisticated development methods to elude advanced antimalware products, many malware strains now use a technique called stalling code in which code execution is delayed and waits for the sandboxed environment to time out. After executable code fails to do anything malicious within a predetermined amount of time, many sandboxed environments simply forward the code to its intended destination. Vendors are working on ways to combat this problem, but it's a useful reminder that when it comes to malware detection, there is no panacea.

Consequently, if an organization has made the decision to purchase an antimalware product that relies on sandboxing, the wise security professional may consider whether completely scrapping the organization's signature-based malware detection capabilities is prudent until the sandboxing concept has fully matured. In fact, security administrators may consider implementing a hybrid approach where more traditional signature-based technology is placed at the enclave, while the more advanced malware detection technology resides a hop or two behind the network boundary. Whichever path is chosen by the security administrator, vigilance should be considered paramount as network attackers are constantly scheming and developing methods for defeating even the most advanced of security configurations.

About the author:
Brad Casey holds an M.S. in information assurance from the University of Texas at San Antonio and has extensive experience in the areas of penetration testing, public key infrastructure, VoIP and network packet analysis. He is also knowledgeable in the areas of system administration, Active Directory and Windows Server 2008. He spent five years doing security assessment testing in the U.S. Air Force, and in his spare time, you can find him looking at Wireshark captures and playing with various Linux distros in VMs.

This was last published in September 2013

Dig Deeper on Endpoint protection and client security