Published: 01 Feb 2011
The malingering debate over the viability and lifespan of penetration testing as an art form and penetration testers as a species is getting tiresome. Tiresome because those doing the arguing generally confuse terms, juxtapose vulnerability management with pen-testing, and generally don't understand what white-hats do during an enterprise poke-and-probe.
Let's get it straight once and for all: Pen-testing is not dead.
Some vendors and expert types would like you to believe that and will try some Jedi mind-tricks to convince you -- for only a second, hopefully -- that you can, for example, automate penetration testing. You can't.
Automated scans are great and are the center spoke of vulnerability management programs. They help with asset discovery and generally are good at telling you what machines are lacking which patches and if you've got a cockeyed configuration or two. But that's not a pen-test, and too many companies are confounding that as a pen-test.
Pen-tests are conducted by people who are contracted to infiltrate your organization and hammer away until they get in. They assume the profile of an attacker and use the same tactics and technologies a persistent attacker would -- and that includes vulnerability scanners, commercial and homegrown exploit tools, social engineering, bolt cutters, lockpicking kits, glasses and a fake nose, and lots more. And that's generally preceeded by a bit of reconnaisance against the target.
Usually, pen-tests are carried out in context of consequences to the business. At the end, you got a pretty hefty document illustrating how Swiss-cheesy your network is and sometimes for added measure, you're handed the list of customer data you challenged the pen-tester to dig up from your unbreakable systems.
It's an eye-opening interaction. It's also expensive and usually reserved for the biggest and most resourceful of enterprise IT organizations. And some of that it is to blame for all this pen-testing-is-dead nonsense. So are regulations such as PCI DSS, which mandates annual pen-tests done by "a qualified internal resource or qualified external third party" and against the network and applications. As one of our resident experts Diana Kelley points out in a recent blog post: "Show me where in there it says that the test should simulate an actual attack? How about where it says the test should be high quality? Or that it shouldn't leverage automation almost exclusively? Nope. If you're doing the testing solely to satisfy the requirement, your decision will be driven probably by economics within the parameters of the audit standard."
There's the rub, and it goes back to another tiresome security meme that companies prioritize compliance and think they're simultaneously secure. PCI's mandates are a minimum standard, yet organizations extrapolate that to "since we passed the audit and earned certification, aren't we secure too?"
Pen-testing isn't dead. But it is changing. David Kennedy, a corporate security manager, summarized a recent BSides Atlanta presentation he and Eric Smith did on the evolution of the art form on his SecManiac blog. From an economic perspective, companies are finding it tough to resist automated pen-tests. He wrote: "How do you go against a cheap vulnerability scan penetration test to something that will cost significantly more than that and be done right. Businesses don't understand the difference, they just go with the cheapest buyer, they don't know what they are about to purchase sucks."
Vendors such as Core Security and Rapid7 (Metasploit) are doing their best to automate the process by converging vulnerability scanners and exploit frameworks in order to turn your IT guys into pen-testers. Val Smith, founder of Attack Research, wrote recently on the Carnal 0wnage blog that organizations have a vested interesting in conducting a low quality test in order to pass a PCI audit, for example. Smith wrote: "Therefore hiring people who can emulate real attackers is overkill, too expensive, and likely to produce a test in which they fail, requiring a costly corrective action plan. Since these customers are the ones paying the money, they are what is driving and will continue to drive the industry."
Automation, however, removes the human element that distinguishes penetration tests. Automated attacks don't test the effectiveness of awareness training and whether your admin will spill your password with some gentle coaxing. Automation won't think on the fly and understand your business and conduct attacks in that context.
Pen-testing may be changing, and more companies might be cheapening out on them. But they're not dead -- they're not even on life-support.
Michael S. Mimoso is Editorial Director of the Security Media Group at TechTarget. Send comments on this column to email@example.com.