Information Security

Defending the digital infrastructure


Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Bruce Schneier and Marcus Ranum debate the necessity of penetration tests

Pen tests identify your organization's weaknesses. Bruce Schneier and Marcus Ranum debate whether organizations really want to document all the ways networks are insecure.

Is penetration testing worth it?

Bruce Schneier

Point There are security experts who insist penetration testing is essential for network security, and you have no hope of being secure unless you do it regularly. And there are contrarian security experts who tell you penetration testing is a waste of time; you might as well throw your money away. Both of these views are wrong. The reality of penetration testing is more complicated and nuanced.

Penetration testing is a broad term. It might mean breaking into a network to demonstrate you can. It might mean trying to break into a network to document vulnerabilities. It might involve a remote attack, physical penetration of a data center or social engineering attacks. It might use commercial or proprietary vulnerability scanning tools, or rely on skilled white-hat hackers. It might just evaluate software version numbers and patch levels, and make inferences about vulnerabilities.

It's going to be expensive, and you'll get a thick report when the testing is done.

And that's the real problem. You really don't want a thick report documenting all the ways your network is insecure. You don't have the budget to fix them all, so the document will sit around waiting to make someone look bad. Or, even worse, it'll be discovered in a breach lawsuit. Do you really want an opposing attorney to ask you to explain why you paid to document the security holes in your network, and then didn't fix them? Probably the safest thing you can do with the report, after you read it, is shred it.

Given enough time and money, a pen test will find vulnerabilities; there's no point in proving it. And if you're not going to fix all the uncovered vulnerabilities, there's no point uncovering them. But there is a way to do penetration testing usefully. For years I've been saying security consists of protection, detection and response--and you need all three to have good security. Before you can do a good job with any of these, you have to assess your security. And done right, penetration testing is a key component of a security assessment.

I like to restrict penetration testing to the most commonly exploited critical vulnerabilities, like those found on the SANS Top 20 list. If you have any of those vulnerabilities, you really need to fix them.

If you think about it, penetration testing is an odd business. Is there an analogue to it anywhere else in security? Sure, militaries run these exercises all the time, but how about in business? Do we hire burglars to try to break into our warehouses? Do we attempt to commit fraud against ourselves? No, we don't.

Penetration testing has become big business because systems are so complicated and poorly understood. We know about burglars and kidnapping and fraud, but we don't know about computer criminals. We don't know what's dangerous today, and what will be dangerous tomorrow. So we hire penetration testers in the belief they can explain it.

There are two reasons why you might want to conduct a penetration test. One, you want to know whether a certain vulnerability is present because you're going to fix it if it is. And two, you need a big, scary report to persuade your boss to spend more money. If neither is true, I'm going to save you a lot of money by giving you this free penetration test: You're vulnerable.

Now, go do something useful about it.

Marcus Ranum

CounterPoint Pen testing is a great idea--if you're a pen tester. Other than that, I think there are serious problems with the concept. Sure, pen tests give you a comfort factor, but I suspect most of their value is in keeping auditors off your back or showing clueless managers that, "Hey! Someone really can break into our system!" That represents a lot of money, time and effort spent on appeasing the clueless. With recent standards and legislation such as PCI and Sarbanes-Oxley, we have what amounts to the Pen Tester Permanent Employment Act.

My mother taught me everything I needed to know about computer security back in 1969 when she asked, "If all your friends were jumping off a cliff, would you jump too?" Just because a lot of people are doing something doesn't make it smart. The problem with pen testing is that it doesn't measure what people want to believe it measures. Gary McGraw likes to refer to pen testing as the badness-ometer--it's a test that registers at one end of the dial Your Network Stinks, and at the other end, We Don't Know.

A more logical way of looking at it is simply that pen testing tries to prove a negative--namely, there are no holes in the system. Any student of logic knows you can't prove a negative; what you can prove is a positive: "Our pen tester doesn't know any way to get in." What you're doing is paying a pen tester a hefty amount to evaluate how good they are.

The only useful outcome of a pen test is the worst one: The pen testers walk in and demonstrate, conclusively, that system security is horrible. Then you've got a 50-50 chance you'll end up with a mandate to fix it. Here's the sad fact: Organizations with poor security already know it, and it is not going to be improved a great deal by having an outsider show up and point that out.

So what's the realistic alternative to pen testing? It's obvious: Have a good security design, and then verify that it is in place and working correctly. If your management wants to hire outsiders because they don't trust you, or they think you're stupid, hire outsiders to review your security design and help you improve it; then you'll actually have something to test. Isn't that a bit more scientific and logical? Your security design is your plan; then you validate your implementation against the plan, note deviations, and reassess. The pen testing approach is to look at your network as a great big unknown, from which you try to derive clues using ping sweeps and port scans. I've got bad news for you: If your network is so uncontrolled that the only way you can figure out what's on it is by scanning, then your badness-ometer is already pegged on Stinks. All you are going to find is large, uncontrolled tracts of TCP/IP swamp land, great unknowns populated with backdoor wireless access points, keylogger-infected laptops and wide-open hosts.

That's the part, I think, that scares me the most about pen testing: It is not a substitute for knowing what should and shouldn't happen in your network. Pen testing is just a revisiting of the old philosophy of "penetrate and patch"--take something fundamentally flawed and keep adding more layers of duct tape and bandages to it, and eventually the flaws will be cured. It doesn't work.

Fundamentally, most security problems are a result of poor design, and it's impossible to get a good design by taking a bad one and throwing away all the bad parts. A pen test helps you identify a few bad parts, but that's an endless effort-sink; there will always be more. That's bad news for you--and good news for your pen tester.

Send comments on this column to

Coming in May:
Is Big Brother a big deal?

Article 12 of 15

Dig Deeper on Penetration testing, ethical hacking and vulnerability assessments

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

Get More Information Security

Access to all of our back issues View All