Bruce Schneier and Marcus Ranum debate the necessity of penetration tests


This article can also be found in the Premium Editorial Download "Information Security magazine: Compliance vs. security: Prevent an either-or mentality."

Download it now to read this article plus other related content.

Marcus Ranum

CounterPoint Pen testing is a great idea--if you're a pen tester. Other than that, I think there are serious problems with the concept. Sure, pen tests give you a comfort factor, but I suspect most of their value is in keeping auditors off your back or showing clueless managers that, "Hey! Someone really can break into our system!" That represents a lot of money, time and effort spent on appeasing the clueless. With recent standards and legislation such as PCI and Sarbanes-Oxley, we have what amounts to the Pen Tester Permanent Employment Act.

My mother taught me everything I needed to know about computer security back in 1969 when she asked, "If all your friends were jumping off a cliff, would you jump too?" Just because a lot of people are doing something doesn't make it smart. The problem with pen testing is that it doesn't measure what people want to believe it measures. Gary McGraw likes to refer to pen testing as the badness-ometer--it's a test that registers at one end of the dial Your Network Stinks, and at the other end, We Don't Know.

A more logical way of looking at it is simply that pen testing tries to prove a negative--namely, there are no holes in the system. Any student of logic knows you can't prove a negative; what you can prove is a positive: "Our pen tester doesn't know any way to get in." What you're doing is paying a pen tester a hefty amount to evaluate how good they are.

The only useful outcome of a pen

    Requires Free Membership to View

test is the worst one: The pen testers walk in and demonstrate, conclusively, that system security is horrible. Then you've got a 50-50 chance you'll end up with a mandate to fix it. Here's the sad fact: Organizations with poor security already know it, and it is not going to be improved a great deal by having an outsider show up and point that out.

So what's the realistic alternative to pen testing? It's obvious: Have a good security design, and then verify that it is in place and working correctly. If your management wants to hire outsiders because they don't trust you, or they think you're stupid, hire outsiders to review your security design and help you improve it; then you'll actually have something to test. Isn't that a bit more scientific and logical? Your security design is your plan; then you validate your implementation against the plan, note deviations, and reassess. The pen testing approach is to look at your network as a great big unknown, from which you try to derive clues using ping sweeps and port scans. I've got bad news for you: If your network is so uncontrolled that the only way you can figure out what's on it is by scanning, then your badness-ometer is already pegged on Stinks. All you are going to find is large, uncontrolled tracts of TCP/IP swamp land, great unknowns populated with backdoor wireless access points, keylogger-infected laptops and wide-open hosts.

That's the part, I think, that scares me the most about pen testing: It is not a substitute for knowing what should and shouldn't happen in your network. Pen testing is just a revisiting of the old philosophy of "penetrate and patch"--take something fundamentally flawed and keep adding more layers of duct tape and bandages to it, and eventually the flaws will be cured. It doesn't work.

Fundamentally, most security problems are a result of poor design, and it's impossible to get a good design by taking a bad one and throwing away all the bad parts. A pen test helps you identify a few bad parts, but that's an endless effort-sink; there will always be more. That's bad news for you--and good news for your pen tester.

Send comments on this column to feedback@infosecuritymag.com.

Coming in May:
Is Big Brother a big deal?

This was first published in March 2007

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: