Marcus Ranum: Joel, I'm sure you know we can't get through this without talking about advanced persistent threats (APT) and malware. What's your take on that topic? I know you're a real malware maven; are there different levels of the state of the art in malware? Is malware evolution slow, or rapid? What's going on out there?
Joel Yonts: Marcus, just the mention of the term advanced persistent threat (APT) has such an impact on practitioners in our field. As you know, these reactions vary from sheer terror to the extreme skeptic that says APT is a term invented by marketers to sell more gear. Since the term is so inflammatory, I will reserve my thoughts on that topic for a moment and focus on the malware problem. Malware comes in many levels of sophistication. On one end of the spectrum, we have malware that can shut down a nuclear facility (Stuxnet) by reprogramming low-level controls. On the other end of the spectrum, I received a malicious spam attachment the other day that had a poorly constructed executable that after hours of trying, I still could not get it to infect my PC. There are many levels of malware in between these two extremes, with the most prevalent mid-level threat being the banker Trojans and keyloggers that are used by identity thieves. Other examples of sophisticated malware include malware that incorporates advanced stealth techniques such as rootkits, covert command-and-control, anti-analysis, and anti-forensics techniques.
When we consider the APT question, however, we must examine not only the malware, but also the actions and coordination of the threat actors behind the malware. Often in advanced threats, the delivery mechanism involves advanced social engineering that specifically targets organizations, or in some cases, individuals. Also, actions taken after a system is compromised may also be a barometer for determining the level of sophistication. In less sophisticated attacks, the malware foothold may be used to steal bandwidth and computing power to send spam and possibly denial-of-service a target. On the more sophisticated attacks, the malware foothold may change rapidly from a malware infection to a full blown network intrusion where attackers move quickly through a compromised network, setting up exfiltration of sensitive data and planting additional remote access backdoors. This last point of backdoors is one of the ways APT has earned the reputation of being persistent. I have heard many incident responders comment on the extreme difficulty of containing and eradicating advance threats within their network.
Marcus: (Wondering aloud) Maybe the 'P' in APT should stand for "permanent."
Joel: There is also a great deal of segmentation and specialization in the malware underground. Specialized services have emerged that focus on specific components of the malware lifecycle. Examples include exploit developers, payload developers, packing and anti-analysis wrappers, packagers, botnet masters, spammers, list (target) brokers, exfiltration agents, underground data forums, and money mules.
This segmentation has driven a maturing process that translated into more sophisticated malware. The other aspect that is driving change in the malware space is a change in threat actors. In years past, malware advancement was primarily being driven by identity thieves, spammers, and possibly anarchists. Today, nation states have an increasing role in cyberthreats. While it is very difficult to pin attribution as state sponsored, I believe there is a mountain of evidence that points to nation state-developed malware. So to answer the APT questions directly, I don't believe in an APT, but rather APTs do exist and some (but not all) are state sponsored.
Marcus: It seems I’m constantly on conference calls with worried executives asking, "What should we do to prepare for APT-style attacks?" I can't come up with any simple answers and usually my response is something along the lines of: "The reason APT is a problem is because you're dealing with attacks that are designed to defeat security systems, and that are being adapted constantly. So, if you could buy an off-the-shelf anti APT product tomorrow, the APTs would be adapted to using techniques that would defeat it in a month or two. What you need to do is respond to specificity in attack with specificity in defense--you need an analytical capability that is grounded in understanding what your networks should/shouldn't be doing and then looking for deviations. You can't buy a product that replaces that analysis."
First off, Joel, have I got that timeline right? Would it be weeks, months, or years? And what do you tell your peers who are worried about APT attacks?
Joel: You are spot on! Advanced attacks do adapt quickly to overcome current state defenses. This adaptation often occurs in the hours-to-weeks timeframe, but I have no doubt some would spend years perfecting the ideal attack, after all they are very persistent. Echoing your sentiments, there are no "APT modules" that can be installed. To win it takes good people armed with the right knowledge and tools. First, advanced attacks must be identified early.
This is where the intimate knowledge of the environment that you spoke about comes into play. Those monitoring the technical environment must be able to spot deviation from the baseline and quickly identify anomalous network activity and system changes. Verizon did a great study where they examined case details from a large number of data breaches and found that on average it takes months for a company to become aware that they have been compromised. Additionally, they found that, in most cases, evidence of the intrusion was clearly present in the company's log files. Obviously, this is not a recipe for success. A well-trained operations team armed with tools to assist with collecting, correlating, and processing security event data is the first step in surviving advanced attacks.
Next, it is critical to have a well-trained incident response (IR) team. Usually you only have one shot at responding to an attack appropriately, and the first few hours can mean the difference in a near miss and a full data breach. We could spend a whole conversation just on the topic of building a good IR team, but in general, the team needs to have the same intimate knowledge of the company's technical and business environment, a solid understanding of the IR process, and at least a working knowledge of attacker techniques and digital forensics.
For those heavily targeted organizations, stronger forensics skills and onsite malware analysis capabilities are also a must. Also, these recommendations assume the foundations of information security are already in place. If the basic blocking and tackling is not in place, the organization will be so inundated with low- to medium-level attacks they will spend their days swatting at issues while the sophisticated attackers are pipelining their data out their front door. So, in a nutshell, doing everything right will get you 90 percent there, but in the end some advanced attacks will get through, and having the right team ready to respond is critical. For some reason, I am getting a mental picture of a gunfight at high noon.
Marcus: I'm really worried that the trend seems to be cost-cutting and everyone I talk to is strapped for personnel resources (which is weird given the high unemployment, isn't it?) How were you able to sell management on being able to build a team and keep them? It seems to me that for the longest time whenever one developed a good analyst, they'd get snapped up by a consultancy and it was back to square one. How did you build your team? What are the traits you'd say make up a good IR team analyst?
Joel: That is the toughest question yet! The answer really varies depending on the business environment, leadership personality and the chemistry of the team. So, while there is not a one-answer-fits-all, there are certainly a few elements I believe are common in successfully gaining buy-in.
Success comes when a security program is relevant, business aware, communicates using business language, and has established a balanced security posture. Combined I believe these build credibility with senior management and credibility usually equates to funding and a seat at the decision table. Of the previously mentioned elements, I fear that relevance may be the element most often missed. In order to be relevant, the program needs to have its finger on the pulse of the threat landscape and to be able to build business-friendly strategies that reduce real risk.
Relevance does not come from chasing media buzzwords and compliance standards. Interestingly, relevance is also a key to engaging the security team and retaining talent. I am also a realist, so the right work environment and good pay are also important. In my current situation I am very blessed with great senior leadership and with a great team. This is important! Having the right people sets the whole program up for success.
To answer your question about how to build a good security team, I look for talent that has a broad technical experience base, passion for learning, and individuals with unwavering integrity. Drilling into these deeper, I believe the best security analysts haven't always been security analysts. Rather individuals who have served many different technical roles before transitioning to security bring with them a contextual awareness that is very difficult to build artificially. Next, security evolves at a lightning pace. The team’s skills need to keep pace. Continual learning is a fact of the job, so having individuals who get their kicks from learning creates the perfect marriage of people and task. Last, the security team has a tremendous amount of access across the enterprise. Knowing that the security team has the integrity to not abuse these privileges is critical.
Marcus: It's been great talking with you, Joel--your attitude always cheers me up. Let me just wrap up with one final question. I know you are a do-it-yourselfer (and so are many of the people who'll be reading this). What would you say, off your head, are your top five tools that you use in your antimalware work?
Joel: Marcus, the feeling is mutual. I always take the opportunity to hear the world according to Marcus!
As for tools, to understand the tools of the trade we must first understand the trade a little better. There are two main approaches to malware analysis. The first is dynamic or behavioral analysis. This method usually requires less malware analysis specific skills and can be a good approach to understanding the artifacts or indicators that are associated with the execution of a malware sample.
The combination of Process Explorer and Process Monitor by Microsoft Sysinternals coupled with the packet analyzer Wireshark really gives anyone with a good IT background the capability to do basic malware analysis.
With these tools, a responder can paint a reasonably complete picture of the network, filesystem and registry indicators that can be used to identify an infection.
Fully mapping the payload and capabilities of the malware, however, requires examining the internal instructions of the malware. This approach to malware analysis is often referred to as static analysis. For this deeper level of analysis OllyDbg debugger and IDA Pro disassembler are industry staples and really give an analyst the nitty-gritty of the sample. Be forewarned, you will need to dust off your x86 assembly language skills to play in this arena. There are a large number of other very useful and time-saving analysis tools, but I would say these would make my top five and they are used by nearly everyone in the malware analysis world.
Marcus: It has been great chatting with you and I look forward to the next time we cross paths.