The challenges of incident response plans and proceduresDate: Jan 21, 2009
Most people who respond to computer breaches are engineers, and according to Mandiant's Kevin Mandia, they bow down to one god: accuracy.
Once an organization loses sensitive or personally identifiable information, the clock starts, and a company has a limited time to disclose the breach. So what's more important: time or accuracy? Kevin Mandia explores the argument and reveals five of his top incident response challenges.
Read the full transcript from this video below:
The challenges of incident response plans and procedures
Kevin Mandia: I don't how many of you respond to computer security breaches, but most people that do, they're engineers, and they bow down to the God of accuracy and they don't get anything done fast because of that. It's kind of a constant tug. However, the minute an incident occurs and you've lost private data, PAI data or non-publicly available information, however you refer to it, a time ticker starts.
In the state of California it's 10 days from the moment you know and confirmed it to the moment you disclosed. The state of Florida it's 45 days. My troops would take 500 days on one case, just reviewing the malware and seeing all the stuff it does. So you have this God of accuracy racing the God of speed and you need opinions and I hate to say it, but most people it's complicated. The engineers will always say, well it's possible that they lost card numbers, but I'm just not sure.
At the end of the day we get asked, did we lose card numbers or not? Most responders, if you do this, what I'd recommend is just create a here's your arguments, yes you lost card data; here's your arguments, no you didn't and let the lawyers decide and if they keep drilling you for an opinion go to lunch.
It's actually complicated and I've tried not to, but every once in a while if there's litigation you need to have an opinion. We have more to that. Then so I wanted actionable stuff and I've got ten minutes now, some actionable things we can get out of this lecture today.
Here's the top five challenges we have, and these are overly broad so I have specifics, failure to sign a throat to choke when there's an incident will go over the problems, but you can't have a democracy ordinarily at a large firm or even a small firm, it doesn't work all that well unless the incident is not that big and it doesn't have a lot of impact. Failure to define the win could be a sub to that, failure to recognize who you're actually responding for, bad documentation or poor documentation and effective remediation. Let's go through these real quickly.
The number one thing that I would do as a CEO to make sure we did everything right on an incident is grab a throat to choke and say Bob, Jill, you're in charge, you're it, you do it right, because then no matter what somehow, some way it will get done right.
What I've seen is if you work at a firm of over 20,000 employees, you have 9 business lines, each business line wants to do it themselves and they all try to have this democratic way of doing things, I tell you right now it really doesn't work and there's a couple of reasons why.
One, if I'm business line A and I've been compromised and I need to re-mediate, it's highly probable business line B, business line C need to re-mediate as well, and you need to coordinate among those business lines and the minute they go off into splinter cells re-mediating on the run the remediation does fail.
So you need that one guy making the decisions when you're going to do it, how you're going to do it, who's doing it and kind of be able to push and compel folks otherwise it gets ugly. You have unclear roles and responsibilities. What I mean by this is when there's an incident what happens? You take your proactive security guys and make them respond.
Hey patch management dudes, hey network ops guys and hey firewall guy, come down here and sit in this room, let's get the decision making mass together and sort this out. They got day jobs to go back to in about three days or stuff just keeps piling up, so unless you have a leader in charge that makes it clear, guys this is your number one priority, this is going to be a four-week incident, tough break, we'll try to staff accordingly with bodies from other business units.
Whatever it is, what I've seen is I've shown up to card compromises at tier one retailers and I was the only damn guy in the room after four days. It's not that they didn't care; it's just that the people they were asking to respond to the incident at the firm simply had day jobs to go back to.
It happens and if you don't have a guy that can sit at the top and say you're showing up every day to handle this it gets ugly fast. So here's a couple of things that are what I think is mastering the detect, collect, analyze and re-mediate process for increased capability.
If you work at a firm and you suspect that a machine has been compromised and you can ask or do it yourself, but you can get live response data or get data off that machine in under four hours that's pretty darn good at most companies. If you can get a drive duplicated at a large firm in under two days from the moment you ask, believe it or not that's pretty darn good.
If you can say, hey I need the IDS from the folks in Malaysia and the folks down in Sydney and you can get that in under four hours that's better than what I see. This is timetables for folks dealing with the APT. All the different things, everything is mostly four hours except disk duplication. The thing about disk duplication is a lot of firms fly a guy in, they've made it this black art on how to duplicate a drive.
Analysis, I do believe if you manage technicians and you're responding to an incident you say, hey John, can you look at the sequel server and tell me if it's compromised? If you don't have an answer in two days something is wrong. Either John can't figure it out, I always say you can't have nurses do brain surgery, you can't have John figure this out, I hate to say it.
Or he just went off on tangents, he's got his answer, but he doesn't want to tell you because he's chasing some bright light like a cool keylogger or a rootkit that's fascinating and he's in his IDA disassembler tool having fun. The bottom line is in under two to three days you've got to be done looking at these systems on compromises because what if you have 100 compromised hosts? You're not going to spend 300 days looking at them all.