Face-off: Information security management metrics

When it comes to information security management metrics, are they worth the time they take, or do they simply distract information security pros from the real issues?

In this face-off, Hugh Thompson, Founder of People Security, and Adam Shostack, co-author of The New School of Information Security, discuss whether metrics are necessary.

Read the full text transcript from this video below. Please note the full transcript is for reference only and may include limited inaccuracies. To suggest a transcript correction, contact editor@searchsecurity.com.  

Face-off: Information security management metrics

Adam Trinishack: Hi, I'm Adam Trinishack and I'm here today with Hugh Thompson. This is the SearchSecurity face off and today we're going to be talking about security metrics. So, let me throw a slightly controversial statement your way.

Hugh Thompson: Throw it, man. Throw it.

Adam Trinishack: Which is we spend a lot of time and energy focused on risk metrics and threat metrics and measuring the attack landscape. And we should get focused on security operation because that's the thing that we can control and that's the thing that we should be measuring.

Hugh Thompson: Okay, let me just say this, before I brutally attack that statement, tell me what you mean by operations measurement. So you're talking about effectiveness?

Adam Trinishack: What I mean is effectiveness and outcome. So how quickly are you getting your patches out? What's the coverage? Are your systems domain joined? Do you have password policies in place? Do you have logs? Measure these things so you can actually assess the security of your organization. You can assess the things that you might have an opportunity to do better. Because trying to measure the events persistent threat, what are you measuring? What are you going to change? Well, advanced persistent threats are 20% more persistent this month, Bob.

Hugh Thompson: I agree totally that we need to focus on decision oriented metrics and not metrics that are just noise. And I think you're right, I think there are some threat based metrics that are just noise. We can't do anything about it, we can't change it, but there's an important reason to measure threat and threat change. So if I'm in charge of security for a group or I'm responsible for security for an organization, I need to message up the chain, what the importance is of what I'm doing and how I'm making the situation better than it was. Which ties back into operational metrics. So I have to be able to paint this picture of the threat that's out there. Some of it we don't know, some of it we do know. Some of the actions that we've taken in the past, but also how we plan to incrementally improve to face this ongoing threat. Like we may not have anything today to deal with the threat of social networking and people talking about information online, but if we can measure that threat in some way, we can at least plan for budgetary perspective this is something that we're going to have to deal with.

Adam Trinishack: So I'm going to propose an alternative.

Hugh Thompson: Okay, good, good, good. I'm open to alternatives.

Adam Trinishack: Here's what we should be doing. We should be looking at outcomes. We should be looking at where things are going wrong. The data lost DB, for example.

Hugh Thompson: Yeah, great project.

Adam Trinishack: Has all sorts of information about what's going wrong. What if we could get a little more information about what really happened at that organization? So we could actually plan and not say, "There's bad guys out there. The threat environment is changing." But we could say, "These companies in our industry who are patching 20% better than us experience 30% fewer incidents." If we could actually give our management that sort of message and say, "Here's the impact that this is going to have on the outcome.", we would get the budget no questions asked.

Hugh Thompson: So I think there's two things though. I would love to live in a world where we could get that data, message all the risks that we care about and show specific case studies and I think we are at the point of a maturity that we do have some of this data at least. So some things we can sort of transfer into this historical data. Company x has done this, company y has done this, we want to be more like company x than company y, but there's also this qualitative element of threats, almost a narrative of what's going on in the threat community, that we can't quite get our hands around from a number perspective. We don't know if it's a 28 or 56 or an 82, but we know it's something that we have to deal with in the future. So from a metrics perspective we may not be able to measure that but we have to be in tune with some of those changes.

Adam Trinishack: You know, that sounds to me a lot like the black swan argument.

Hugh Thompson: Okay, the incredibly unlikely event of recurring.

Adam Trinishack: Yeah, it's we have in our back yard a set of swans and ducks and geese. But we don't know how many. We don't know how much they weigh. We don't know how many eggs they're making. We get focused on the black swan because the black swan is exciting.

Hugh Thompson: I don't know man. In some ways I can see what you're saying. But I think security and a lot of the threats that we're talking about, we see the black swans. Like, dude, we have two black swans just kind of walk by the window. We're seeing they're out there, but we just don't have any data to measure how many of them there really are. Should we defend or should we take some geese taekwondo?

Adam Trinishack: You just said there were two of them.

Hugh Thompson: Right. But how many more in the flock? We don't know, man. So I think that some of these elements, like we're seeing increased risks in certain areas of very, very tailored phishing attacks for example, they may not have yet hit the company. We may not have enough data from other companies that have dealt with this in an effective way so we don't have the exacts or historical metrics to measure. But we know this is going to be a big threat for us in the future and it's something that we have to put on managerial radar. And just to educate them to the point that we don't know how much we can reduce this threat by putting in control x, but we know it will help to some extent.

Adam Trinishack: Sure and managing a company is all about risks.

Hugh Thompson: Yeah, it's all about risks.

Adam Trinishack: You don't know what sales are going to be next quarter. You take steps to try and deal with them. But there's a lot of people who I talk to who look at security as always about the black swans and never about measuring.

Hugh Thompson: As an industry I can say that we've been in the black swan from a metrics perspective, for a really long time. I think we can see some of the advantages we have today, which is an openness and a willingness of some countries to really share their data.

Adam Trinishack: Some transparency.

Hugh Thompson: A little bit of transparency that's now happening in the industry. Definitely encourage that to continue. But also realize there's an element of risk that we can't fully understand yet, and educate management about that too.

Adam Trinishack: I think you're right and I'm hopeful that Howard Schmidt is going to say some things around transparency and government data about security. Because I think it would be a big step forward.

Hugh Thompson: It's a brave new world man.

Adam Trinishack: I'm not sure that's the analogy we want.

Hugh Thompson: Yeah, good point.