No one wants to talk about successful hacker attacks, but they are a part of life and they come with the territory of being an information security professional. Most security folks don't think about these types of incidents because they are unpleasant and there are lots of other tasks to focus on, so this discussion usually falls to the back burner.
I suggest that stance be revisited because the difference between the "hero" and the "goat" is how the inevitable incident is handled. Many security professionals end up looking for another gig because they bungled the response to an incident and didn't provide investigators or law enforcement officials with enough information to figure out what happened.
Proper post-incident due diligence requires a different mindset. A lot of folks consistently do penetration tests, threat modeling exercises and other techniques to "think like a hacker" and figure out how they would break their own systems if given the chance.
In order to correctly develop an incident response plan, it's important to think like an investigator. In the event of an attack, breach or other situation, what data would be needed to piece together a true picture of what happened and why, and bring a perpetrator to justice? What processes should be in place to ensure important tasks don't get missed in the heat of battle?
Logs: The investigator's best friend
First, make sure the proper log files are kept for all applicable systems. It's hard to investigate without any data, so the first shift in mindset needs to be around logging. This doesn't just mean on firewalls and IPS devices; it means in databases, on application servers and within data centers. Logs should be taken and stored everywhere possible.
An important consideration is how long to keep the log files. There is no generic answer to this, except to know that having more data is better than having less data. It's never clear when a breach first happened; if the logs were rolled and data was over-written, it's gone for good, which means the job of the investigator just got a lot harder.
Effective logging creates a huge amount of data, which is why hopefully a purpose-built log management platform was under the tree this past holiday season. These products are built for high-speed insertion of many log records from many devices. They can also do some analysis and limited correlation of records, but that isn't the point when you are dealing with investigations.
Evidence -- Will it stand up in court?
Even if your log files make the investigators happy, bringing the perpetrator to justice is another game altogether. The term "opposing counsel" will inevitably crop up, and when it does, it won't be a happy day. These are the attorneys who will try to keep their client out of the big house by poking holes in all of the evidence.
For a successful prosecution, it's critical to maintain a chain of custody for the evidence, as well as store it in a forensically clean way. This involves signing, sequencing and encrypting each log record. Yes, it's important to do these cryptographic operations to prove that the logs have not been tampered with, thereby helping to ensure the perpetrator won't get off based on a legal technicality.
Finally, log data should be physically separate from the devices. The first thing a bad guy does is access the log files on a compromised device and cover his tracks. If the logs are moved to a separate, secure storage platform, the the bad guys will have to go through a lot more hoops to get rid of the evidence.
Bringing in the big guns
The final consideration when adopting the forensics mindset is to know when to bring in the big guns, such as lawyers, law enforcement and the like. This should be defined clearly in an incident response plan, which again is one of the key documents and processes in every security professional's arsenal. The worst time to figure out that your incident response plan doesn't cover all the potential incident scenarios is during an actual incident.
Take the time now to look over your response plan. Make sure it defines when the general counsel and other senior personnel should be notified. It's also important to make sure it's clear when a team of forensic investigators will be brought it. Finally, think about if and when to consult law enforcement. It's not a good idea to make this stuff up in the middle of a crisis. That will result in lots of second-guessing and discomfort -- trust me on that.
To wrap-up, in order to make sure the investigators have what they need to do their job, it's important to keep as much data as possible and have a well-scripted, practiced incident response plan to ensure the right people are consulted at the right time. No one wants to consider that fateful day when his or her environment is compromised. But if the proper processes aren't put in place early, the perpetrators will never be brought to justice.
About the author:
Mike Rothman is president and principal analyst of Security Incite, an industry analyst firm in Atlanta, and the author of The Pragmatic CSO: 12 Steps to Being a Security Master. Rothman is also SearchSecurity.com's expert-in-residence on information security management. Get more information about the Pragmatic CSO at http://www.pragmaticcso.com, read his blog at http://blog.securityincite.com, or reach him via e-mail at mike.rothman (at) securityincite (dot) com.
