Security School

Browse Sections


Get started Bring yourself up to speed with our introductory content.

Essential security analytics technology for advanced malware detection

Josh Sokol reviews the security technologies needed to support a successful security analytics program focused on advanced malware detection.

A few years ago, a company tried to sell me its latest and greatest product for detecting and preventing malware...

from infecting my corporate systems. I politely declined, explaining that my company, full of engineers and other smart people, couldn't possibly be affected by the scores of malware that plague the rest of the Internet. Besides, I told the vendor, we already have a corporate antivirus product designed to prevent these types of issues.

The antimalware software that we've hung our hats on for so long provides little more than an illusion of security.

Fortunately for me, I tried out a demo unit anyway, setting it up to capture our edge Internet traffic off a Switched Port Analyzer port. What completely shocked me was that, as soon as I turned on the appliance and logged in, I saw how bad our problem really was. We immediately began seeing malware detections from more than a dozen systems in the company, even though they all had antivirus installed and were updated with the latest virus definitions. Each of these infections was busily making command-and-control callbacks to servers halfway around the world, and likely had been for a while: We just hadn’t known it. Some of the traffic appeared to be fairly benign link fraud, but other malware was sending encrypted data that we were unable to decipher. Regardless, it was clear that we had a problem and that something needed to be done. Thus began my journey into the world of security analytics.

Malware affects us all, no matter what defenses our organizations have in place. It is a stealthy and sophisticated threat, and the antimalware software that we've hung our hats on for so long provides little more than an illusion of security.

In this article, we'll discuss the various types of products needed to detect and prevent contemporary malware, advanced persistent threats (APTs), zero-day exploits and more, and cover ways to feed data into a security analytics program to create a new, broader perspective on the threats that your organization faces.

The first, and arguably most important, technology that supports a malware-centric security analytics system is a dedicated advanced malware prevention product like the one I described above. In my case, FireEye was the vendor that I selected, based on its unique use of virtualization technology, but vendors including Damballa, Bit9 and many others offer similarly compelling products.

FireEye's threat-prevention platform analyzes traffic in real time, and restricts malware within a virtual machine for further analysis. The product is still able to look for common malware signatures, but it is also able to base detection on the heuristic behavior of the system. This is particularly important in identifying APTs and zero-day attacks for which signatures simply do not exist.

One drawback with the FireEye product is that it detects malware only on systems connected to the network that the appliance is covering. That's a huge gap; many mobile devices would go unprotected. That is where the agent-based approaches of companies like Trusteer or Bit9 come in. By installing agents on each of your endpoints, you can protect devices no matter where they are: in the office, at home or on the road.

If having a dedicated malware prevention system won't work for your organization, you may want to take another look at your intrusion prevention system (IPS). I've noticed a large number of IPS vendors building malware detection rules into their products, with some coming close to the functionality that dedicated advanced malware detection vendors offer.

Configuration management is also a key piece of a security analytics program. The idea here is that you should take an inventory of key configurations and executable files on your critical systems (domain servers, application servers, Web servers, database servers and so on) because an attacker will typically try to replace these files with new versions in order to maintain a foothold in your environment. The open source version of Tripwire is a free data-integrity monitoring tool that is excellent and that security professionals have used for a long time.

It may seem a bit strange, but our network-scanning tool plays a large role in our security analytics program. The best way to prevent malware from compromising an environment is with effective hardening. If I can use a network scanner to search for unpatched and outdated systems on my network, I can remediate them before bad guys compromise them. The good news is that because there are so many competing network-scanning vendors, there are plenty of good deals to be had, and the features don't vary much from product to product. There are also at least a couple of free tools out there to perform network scans, including Nessus and OpenVAS, though they have certain limitations compared to the paid tools.

Another important piece of security analytics for threat detection is log management. The idea is to take all of the logs from all of your systems and store them in a centralized and secure location for future processing. When an attacker compromises a system, he or she usually tries to delete any evidence of the intrusion by editing or deleting system logs. Offloading these logs to a central repository ensures that the attacker would have to go to far greater lengths to tamper with them. Also, with centralized logging, it becomes far easier to search and run reports on all of your systems and applications at once.

There are some things worth inspecting in your logs: multiple failed login attempts followed by a successful one, users who normally log in from one IP or location but are suddenly coming from elsewhere, machines connecting to network IP addresses without doing DNS lookups, or connections with large amounts of egress traffic. Any one of these events may or may not be an issue, but a combination could indicate an attack.

Once again, if a commercial log management or security information and event management product isn't an option, there are some great free SIEM tools available. Splunk is like a search engine for your logs, and it can be licensed for free for use with up to 500 MB of logs per day. I've never used this next one myself, but others I know have had success with a free, open source log-management tool called LogStash.

The last piece that I'd highly recommend for any security analytics program is a network analytics tool that is capable of capturing and analyzing flow data from your various networks. This flow data is a summary of the IP addresses, ports, protocols and data sizes of the traffic flowing across your network. Basically, it's everything but the data itself.

Your network analytics tool will allow you to search for patterns in your traffic that were previously hidden. For example, early last year HD Moore posted a blog entry on the exploitation of Universal Plug and Play, or UPnP, Simple Service Discovery Protocol devices on the Internet. Using my network analytics tool, I ran a pattern query for an external source IP address, targeting one of my public IP addresses, using the User Datagram Protocol on port 1900. In 24 hours there were 539 matches for this pattern. The barbarians are indeed knocking at our gates.

Because forwarding flow data is really just a feature on some routers and switches, you need to find a way to capture and view this data. This could be via a dedicated network analysis tool from companies like Solarwinds, NetScout or Lancope, or from one of the log-management tools already mentioned. I selected a tool called LYNXeon by 21CT, based on its ability to do pattern analysis of not only flow data but also other data types. Allow me to elaborate on that.

I did a presentation at several conferences last year called The Magic of Symbiotic Security, during which I described a security ecosystem that fosters integration -- breaking our tools out of their silos and make them work together for maximum effectiveness. Our ultimate goal in security analytics should be making each of the pieces that I mention above work together to get a clear picture of the threats we face. We take the alert data from our malware- and intrusion-prevention systems, which includes information about the malware type, target and source. We also take any information that we have on systems where the file signatures have changed unexpectedly. We take data about the vulnerabilities that exist on the various systems in our environment. And then we take information derived from log files, such as failed login attempts or account lockouts. We do this using whatever means the vendor provides us to access the data. This could be via an API call, an Simple Network Management Protocol alert or even a direct database query. All of that data is fed into LYNXeon. Next, we use Pattern Query Language to find potentially malicious patterns across any or all of these data sets.

Here are some examples of reports that we run to give us better insight into the security of our network:

  • Internal systems connecting to servers in the
  • Internal systems connecting to many other internal systems over a short period of time.
  • Internal systems connecting to external systems for which our malware- or intrusion-prevention platforms have triggered an alert.
  • Internal systems using DNS servers that are not part of our corporate infrastructure.

Additionally, one of the really cool benefits of combining flow data with other data types in my organization is that we are able to create patterns based on incidents at one location, and use those patterns to find similar issues at other locations that don't have all of the same detection mechanisms in place.

The advice outlined above is based on what has worked well for me, and is certainly not a comprehensive list of tools or even tool categories, but it offers a foundation to get you started with using security analytics for threat detection. Establishing a security analytics program at your company will likely not happen overnight, but it most certainly can be done and will undoubtedly prove to be highly valuable in increasing your ability to detect malware throughout the organization. Don't forget to keep good metrics as you progress so that you can show the ROI to management. Good luck!

About the author:
Josh Sokol, CISSP, graduated from the University of Texas at Austin with a bachelor's degree in computer science in 2002. Since then, he has worked for several large companies, including AMD and BearingPoint and spent some time as a military contractor. He is currently employed as the information security program owner at National Instruments. In his current role, Sokol manages all compliance, security architecture, risk management and vulnerability management activities for NI. He has given dozens of security-related talks, including "HTTPSCan Byte Me" at Black Hat 2010. He currently serves on the OWASP Global Board of Directors.

This was last published in January 2014

Dig Deeper on Malware, virus, Trojan and spyware protection and removal

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.