Incident response planningDate: Jun 22, 2009
Jack Phillips, managing partner of security research firm, IANS, talks about how companies can prepare to appropriately handle a security incident. Researchers have been working on ways companies can effectively address security incidents in a coordinated way. In this video, Phillips identifies the typical stakeholders in an incident and when legal should get involved. Security consultant Lenny Zeltser recently explained how to run a well coordinated response in the initial throes of an incident.
Read the full transcript from this video below:
Please note the full transcript is for reference only and may include errors. To report an error, contact firstname.lastname@example.org
Incident response planning
Rob Westervelt: Hi, I'm Rob Westervelt, the news editor of
for joining us. Today we are going to be talking about security incident
response with Jack Phillips. Jack is the founder of research firm IANS.
Thanks for joining us, Jack.
Jack Phillips: You're welcome.
Rob Westervelt: When do you recognize that an incident deserves a full-fledged
Jack Phillips: The folks that have solid incident response plans have a threshold
above which an incident needs to go before it warrants a full response. So
one of the inherent qualities of a great plan is to know when to apply the
plan, and, more importantly, when not to apply the plan. So the
organizations that we follow, and that we watch, tend to spend a lot of
time on those thresholds. The thresholds can determine, at the highest
level, how the organization responds, and then at more of a tactical, a
micro-level, how certain functions would respond depending on the quality
of the incident. So, that question of 'Where's the threshold?' comes up all
the time, and that is what security teams who work with us really are
trying to figure out: where are they on that Bell curve? In other words,
is there a common standard that says that above a threshold, you need to
respond, below a threshold, you don't. Unfortunately, as is the case in
most things in security, it depends. So every organization needs to really
set their own threshold.
Rob Westervelt: Jack, who's typically involved in handling an incident?
Jack Phillips: It's a great question. There's commonly a struggle between three or
four different groups inside of an organization, and those groups reflect
the outside stakeholders, well the outside and inside stakeholders, in a
given organization. So obviously it starts with technology. So, for talking
about some sort of technical incident, some sort of computer-based
incident, it may be coupled with a physical incident. But let's assume for
a second we're just talking about some sort of a digital incident.
Obviously it starts with IT. Then, the next question really is, do you
bring in the messaging folks? The PR and the marketing folks, because, do
you need to tell the world or do you not? And generally that land leads to
Legal. Because Legal will decide, 'Hey, this is above the threshold, or
below the threshold of alerting federal authorities, or law enforcement,
state authorities' in the case that there are industry regulations that
folks may have to adhere to.
Then I think that the final group is really
going to be the board level or executive management because if in fact an
incident would lead to high risk, again back to that threshold, hard to
know what that means. But let's assume that you've determined it's a high
risk incident. Generally, then, it goes to the CEO and/or board level for
determination around what the right incident would be. So, you know, in sum,
it generally starts with IT and then it spreads from there, and that comes
back to the first question, which is whether the threshold may or may not
need those other groups to weigh in on.
Rob Westervelt: When should Legal counsel be informed?
Jack Phillips: Again, case by case, but I would argue that the highest performing
security teams, the highest performing organizations that we watch, have
heavy involvement, is the short answer. So Legal is quite well-versed in
everything from post incident, evidence handling, to what exactly the
various regulations or industry rules are, that that organization is
affected by. I think the answer is you know, you could argue that they
really are the quarterback on the team. When it really boils down to it,
they really are the ones who are going to determine whether there is real
risk or not, and what the response should be. So IT I think loves to
believe that they really are the holder. But truthfully, probably Legal
would be the driver.
Rob Westervelt: Who typically handles the decisions that could
actually affect the
business, like shutting down a server?
Jack Phillips: I think that a proper instant response team has the classic check-
and-balance system. The highest performing models that we have seen are
the ones that really seem to work the best. It's the system where you have IT
on one side, and/or Risk Management on one side, and then you have the
business owner, then maybe the CEO, then maybe a business line manager on the
other side. They are the ones who proxy in for 'Hey, this is going to cost
us real revenue, if you shut down that system, based on this incident.' IT,
on the other hand is saying, 'Yes, however, we may lose a dollar of revenue
there, but we could have ten dollars at stake if we have a real infiltration
or we've really lost data' for example, or whatever the incident is. So
there is a classic check-and-balance, and I think ultimately the CEO and the
board would adjudicate that tension, in, say, 'I believe it's this', or 'I
believe it's that', and make a decision.
Rob Westervelt: So when should you actually make a decision? How much information
should you gather?
Jack Phillips: It's really a great question, and again, I'm not going to be able to
give you an exact answer. But what we have found is that first of all,
you're not going to be able to make a decision with perfect information.
So the longer you can wait to gather more information without facing undo
risk, the better off you are. At some point you do have to make a decision,
and so, back to if you have a well-rehearsed response plan, and you
determine certain thresholds of information that would trigger decisions.
You can almost practice this out. You can almost say, 'If we don't know the
root cause of the incident, if we really don't know the root cause, we
shouldn't take action.' Or, 'if we feel a 50% conviction that we
know the root cause of something, and we know it's going to lead to some
bad things, we should make a decision'. So the common theme that comes out
of everything that we publish and that we write about at our organization
says, in essence, practice your values. You'll never be able to say, 'OK
we're at 50%, or at 60%, just rehearsing values.' That
says really what are we all about, you know, because it comes down at the
gut-level call, at the end of the day.
Rob Westervelt: We hear so much about PCI compliance, yet there are businesses out
there that have had a breach, that were actually PCI compliant. What is
Jack Phillips: The answer is, I think it's really been an over-blown, really cottage
industry, that has emerged around this rule. This is really what it is, an
industry rule, and all kinds of questions about what motivated the rule,
who really was behind this, who is protecting who's behind, if you will,
and so now we've gone through this classic life cycle where we've hit the
peak, and now there's really I think, a retrenchment back, in terms of
spending and attention, from the organizations who are really affected by
this. So, I think they all understand that we will never be one hundred
percent compliant, PCI compliant. As you said, PCI compliance does not
equal good security, and the opposite is true, as well. So the
organizations that we work with will often say, 'We're going to decide on
a good-enough. We're going to decide on a certain budget amount that we are
going to spend on this, we're going to decide on a set of activities on PCI,
and then we are going to stop, and essentially we are going to roll the
dice.' The large organizations are saying, 'Will the PCI council, will the
card companies and the issuing banks truly shut us down if we are as large
as we are? Probably not.' So in some respects, I think that larger
organizations, and then you look at the small SMB market, and I would say
that most Tier 2 and Tier 3 merchants are largely rolling the dice on
this thing. They would say, 'We can spend it now, or we can spend it later,'
and the idea that they would, in essence, lose their license to process
credit cards, truthfully, I don't think that any of them really look at
that, and say that it's a reality. I think that most of them say, 'Look,
I'd rather just wait and see.'
Well, because the cost to be PCI certified,
I think, really for them outweighs the risk, so as we've matured, we've
gotten through a period where everybody said, 'OK, you know, whatever the
cost, we need to be PC compliant.' So they spend some money. And now, the
market has cleared a little and we are on the back side of that maturity
curve. Now folks are saying, 'Maybe there's a limit to how much I'm really
going to be scared by this.' Now the council has done a good job in
tightening up the rules and clarifying things, that it makes it easier, but
it's half-way to the wall, you know. You are just never going to get
complete clarity on this rule. Again, the companies
that we are seeing are just basically saying, 'Here's how much we've
allocated, it's an ongoing part of our business, we are going to go on to
Rob Westervelt: Let's get back to the roles. What role does senior management play
during an incident response, and how do you go about educating them?
Jack Phillips: Two great questions. I think the role comes back to what I have
referred to earlier. It is really being the referee on the tough calls. If
an incident response plan has been well designed, and it has been well
rehearsed, there will be some gut-level calls, and the worst incidents
where senior management simply has to make, in some cases bet the
organization, bet the company decision, they should not get down into the
tactical steps, but rather should only see things that rise up. In the
analogy that I used earlier, where the team says, 'We're 50%. We just can't figure it
out; it's a flip of the coin.' In other words, 'Do we announce that we've
had a breach?' And in all the breaches we've been following and that
everybody is familiar with, I think there is that moment where you say, 'OK,
we're not sure it was us.' There is a balance here. We don't want to
announce too early, for fear that customers might walk away, and that it in
fact may be found that it wasn't our negligence. Conversely, there is the
other side that says, 'Better safe than sorry.' So, I think truly, that the
board and management needs for something to come down and make those
business-level decisions, after everything else has been done.
The second question, of how to educate them, is probably the
part because they are busy folks, they have a lot of things on their
plates, and the security leaders in most organizations get a very short
amount of time, a short window to educate the board and management on the
risks that are faced, and more importantly, their role when something bad
happens. And so, it's difficult. It's very difficult, and you see very
smart CEO's make the wrong choices. The advice that, I guess, I would
provide to security leaders is, 'Focus your leaders on values, on
ultimately making those hard calls. Don't focus them on what happens if
data tapes fall off the truck, and don't get them focused down in the
weeds. Get them focused on, if something happens, you need to be able to
come through and say, 'You know, that this set of stakeholders weighs out
over this set of stakeholders, or this outcome is more important than the
Rob Westervelt: Where does liability fall during a breach? Does it fall with the bank,
the merchant, or the issuer?
Jack Phillips: It's a great question, and there are so many parties now that are
pressing in on the merchant, that are involved in this now, and that's
what makes life difficult for the merchant. Ultimately, you could say the
liability, core liability yes, would sit with the merchant. But at the same
time, there could be a liability I could imagine, with banks who perhaps
pre-announce. So, in the case of Hannaford, it was in fact the
Massachusetts Banker's Association that actually broke the news that a
large grocery retailer in the state of Massachusetts was having some
issues. And had there not been any later evidence against Hannaford, there
might have been some liability in the bank for pre-announcing that. I think
the short answer is that the merchant really is the one who faces most of
the liability with a push-in from, again, there is the media, there are the
banks, there are the credit card issuers, there is federal law enforcement,
there are the regulators, and there are the industry regs, and rule makers,
industry associations. It's a tough world, particularly for the SMB
merchant, for mid-sized retailers who don't feel as the Tier 2 merchant, a
Tier 3 merchant. You know, I think they are fearful, but at the same time
they also are running the math, and they are saying, 'Boy, I don't know
that I am really going to go the distance to make sure that I'm a hundred
percent PCI compliant,' which, by definition, means that you are zero
percent. Either you are, or you aren't. So ultimately yes, it's the merchant,
and it's a difficult situation for him.
Rob Westervelt: Well, Jack, thanks very much for being with us.
Jack Phillips: You are welcome.
Rob Westervelt: And thank you for being with us. For more information on this topic,
you can go to SearchSecurity.com