Security incident response 101Date: Mar 19, 2009
Sometimes the best procedures fail to overcome the stresses in the initial throes of a breach response. Security consultant Lenny Zeltser explains ways to carry out a well coordinated security incident response. The lack of clear leadership and the failure to ask questions can cause an incident response to flounder, Zeltser said. A consultant and member of the SANS Institute board of directors, Zeltser presented his incident response research at the SOURCE Boston security conference.
Read the full transcript from this video below:
Security incident response 101
Interviewer: It seemed like a lot of this is common sense, but it gets lost in the moment, in the initial response to a date of reach for example?
Lenny Zeltser: Yeah. There are two things that are common sense that get lost. One thing is, that as security professionals we often get surprised when organizations don't follow our advice or recommended practices. And the mistake people make, is we offer our advice that assumes they are prepared to a certain stage to take it. When I am trying to figure out, how to offer advice that is common sense in the context of organizations not being ready. The other, of course, common sense issue is that when an incident happens. The situation is very high stress, and therefore people make mistakes under pressure. Because their manager is looking at them, and maybe second guessing everything that they do, and the customers, and the business owners, maybe the board gets involved, and you don't know what to do. Sometimes people are shocked into an action, or sometimes they take too many actions that end up not being appropriate.
Interviewer: One of your major pieces of advice is to get a state of calm in the initial throes of an incident.
Lenny Zeltser: Yeah. I think in many cases, you project how you feel, and when you are the incident handler, the team that is involved in the incident is looking to you and gauging by how you feel, how they should act. So, if you are calm yourself, and you project certain confidence and project certain authority, then all of a sudden the situation that was stressful a minute ago might seem OK. But, of course, the question is, how do you make yourself calm and comfortable. That's why I'm thinking having a little cheat sheet of questions that you ask beforehand. And this way if you walk into a situation that's unfamiliar to you, you feel comfortable because you already know what questions to start asking in the very beginning.
Interviewer: It's not always clear-cut who is in charge.
Lenny Zeltser: Yeah. In many cases you walk into a situation and you don't know, first of all, what the environment is like. And also you don't know who are the people that are affected by the problem, and yes, who's authorized to make decisions. Sometimes, maybe it's you, or maybe it's your colleague, or maybe it's the business owner, or maybe it's the customer. So, yeah, it's one of the first questions that I tend to ask when I walk into a situation, who is in charge? Sometimes I ask that question even if I know the answer, because I want everybody else who is participating in the discussion to hear the answer. Because, what happens if there are, let's say, too many people who are in charge, assumptions are made regarding who does what, and the next thing you know, nobody is doing anything. It's the problem of too many cooks in the kitchen. I want a single person to be in charge. I want that person to know that, and I also want people around me to know that that is the person who is in charge.
Interviewer: The other piece of advice you gave was not necessarily to assume that you know everything, and don't be afraid to ask questions.
Lenny Zeltser: Yeah. For me lately, it's been all about what questions do I ask regardless of the context with whom I'm asking the questions. In incident response, there could be two situations. One situation is you're an outsider. Maybe you're a consultant. I do security consulting and I often times come into an environment that I am not familiar with, and then I have to ask questions. Because I don't want to assume because of my previous experience, that this is what the environment is like. I feel comfortable asking questions when I find that when the people hear you asking the right questions, that there itself gives you a lot of credibility. But, even if you perform incident response within your organization, in many cases you come in, you're not familiar with this team, you've never worked with this department, you don't know much about them. And, another caveat, or another thing to keep in mind, is even if you have responded to an incident before, or used to an assistant admin here, and you think you know the systems, well, maybe they've changed. Don't assume the environment is the way you remember it being. Maybe they've upgraded. Maybe they've replaced Windows with Unix. Maybe they've started hosting a different application in this infrastructure. If you assume you know what's going on, you're just setting yourself up for failure, I feel. That's why I place a lot of emphasis on asking the right questions in the beginning. I think it's a wise use of your time. Too many people in a stressful situation, like incident response, will just start making decisions and taking actions without just taking five or ten minutes, maybe an hour, to make sure they understand the situation.
Interviewer: What are some of the areas involved in understanding the scope of an incident? I understand that it's important obviously to know how data flows through your infrastructure, right?
Lenny Zeltser: When you're trying to understand the scope of the incident, there is a lot that comes to mind. Sometimes when you are in the heat of the moment, you actually don't know what to ask. The things that come to mind at this moment are things like, what systems are affected, what networks are affected. We understand that maybe we know which specific systems have been compromised; we'll try to understand the relationships between those systems and other infrastructure components. Maybe this is a web-server that has log-in credentials to your back-end database, well now all of a sudden the scope of the incident is larger than you thought it was. You're trying to understand the specific infrastructure that is involved, you're trying to understand the dependencies within that infrastructure, and of course you're trying to understand the applications involved. Perhaps more importantly, the data that's part of it, how it flows in and out. Where does the data come from? Maybe it got tainted before it got in. Or, if your system is compromised, where does the data go? Because now the business process that depends on the data may have been compromised as well. My emphasis here is not just on trying to understand the scope of the incident, at the technical, OS and network level perspective, but also expanding it to understand the business process and the business users, and customers that might be within the scope of the problem.