In part 5 of this face-off series, filmed at ISD 09, Bruce Schneier and Marcus Ranum give their takes on the most (and least) effective security metrics. This segment covers the following topics:
- 0:10 Ranum: How to lie to your boss
- 1:14 Schneier: The struggle for good metrics data
- 2:20 Schneier: All about intelligent analysis
About the speakers:
Marcus Ranum is Chief Security Officer for Tenable Network Security.
Bruce Schneier is Chief Security Technology Officer for BT.
Check out other topics in this series:
- Part 1: The future of information security
- Part 2: Social networking
- Part 3: Compliance and security
- Part 4: Cybersecurity coordinator
- Part 6: Audience questions
Read the full text transcript from this video below. Please note the full transcript is for reference only and may include limited inaccuracies. To suggest a transcript correction, contact firstname.lastname@example.org.
Schneier-Ranum face-off part 5: Security metrics
Marcus Ranum: Honestly, I don't think any of these metrics are really worth
anything. Really, what these metrics are, when you go and you do a risk
assessment, you go in and you say, "Well, you got a 3.5". These are just
ways of formalizing lying to your boss because you can't estimate the risk
of some unpredictable thing happening, but when a manager asks for a risk
assessment about something or other going wrong, to me what he's just done
is wave the idiot flag. He's just said, "I want you to give me a
comfortable number because I am too stupid to understand the nuances of the
problem that I have given you", or, "I don't trust you enough, Mr. Security Expert, to go out and
figure out the nuances of the problem, give me a recommendation in writing,
put your career on the line, if you feel like it, on that recommendation
and I'm going to hold you to it. Be prepared to explain it", right? That's
what a security expert does. When somebody goes, 'Is this a bad idea?'. My
job is to sit there and go, "Is it a bad idea? Why is it a bad idea? How is
it a bad idea? Is it a good idea?" You know, make those explanations. Do
not come in there and go, '3.2'.
Bruce Schneier: You know, I'm not impressed with security metrics either. I
mean, what's a good security metric for your network? It's not going to be
how many times you were scanned, because that's nothing to do with you.
It's going to be, what? The number of dollars lost due to fraud? The number
of credit card numbers exposed? I mean, you think about metrics and they
always kind of make no sense.
One of the problems is you just don't have the right sort of environment.
You just don't have good data. You don't know what the rate of fraud is out
in the world, to know if you're doing better or worse than average. So,
it's very hard to get metrics that make any sense at all. The problem is
when you start dealing with very large losses, losses in the millions, and
very small probabilities. Terrorism also falls under this. You're basically
multiplying zero by infinity and if you note any infinity theory,
basically, that equals every number. So, by making very small modifications
on these two numbers, you can make a product be whatever you want.
This is also why ROI models never make any sense. They're fundamentally
made up, because you can make them say whatever you want. So, I think
we're stuck. Metrics are real important, and we're living in a society,
especially in business, where metrics sort of run the world but
insecurely, because you've got smart attackers, because you don't have good
data, because you have evolving threats, because you have rare events,
there just aren't good metrics to hang your hat on. So, I actually agree
with Marcus that intelligent analysis is much better than metrics.