SAN FRANCISCO -- Is penetration testing stupid?
For Brett Hardin of Web analytics vendor KISSmetrics, that may be a loaded question, but the answer seems to depend on what one wants out of a penetration test.
If you're limiting the scope of the [penetration testing] engagement, you're not mimicking the real attackers.
Brett Hardin, KISSmetrics
At Security B-Sides San Francisco 2013 Monday, Hardin gave a presentation titled, Penetration Testing is Stupid, which broke down what he sees as the five questions of penetration testing: What? Who? When? Where? Why?
He used an official definition from ISACA to indicate that pen tests are supposed to test both security defenses and mimic real-life attackers. A former pen tester at Ernst &Young, Hardin believes the typical penetration test largely doesn't accomplish the mimicking aspect.
When and where pen tests happen represent the core of the issue for Hardin. When most organizations hire penetration testers, they lay out the rules of engagement; things such as when the pen tester can attack and what areas their attacks should target. Of course, real-life attackers don't work on a set schedule and tend to take whatever avenue they can to infiltrate a system.
"If you're limiting the scope of the engagement," Hardin said, "you're not mimicking the real attackers."
Hardin addressed the question of "whom" by dividing pen testers into two groups: the average and the best. He described average pen testers as common, cheap and necessary. They rely on methodologies that the field has created, including automated pen testing tools, but rarely find anything that an organization doesn't already know exists.
The best, on the other hand, find the unknown. "The best penetration testers are very different. There is a huge gap between the best and cheapest," Hardin said. "They don't follow methodologies; they invent new attacks. The best actually do mimic the real."
Does this mean that organizations should only hire the best to do their penetration tests? Not really. "At the end of the day, the best are actually overkill," he said.
They types of attacks typically performed by pen testers are one of the few areas where Hardin feels they have accomplished the task of mimicking real-life attackers. This is because pen testers tend to rely on exploiting the same low-hanging fruit as attackers because anything beyond that is generally not needed.
"New exploits aren't needed … because known exploits work," Hardin said. "This is one area where pen testers mimic attackers, as they won't use truly advanced attackers."
Hardin finished with an attempt to answer the all-important question: Why do organizations do penetration tests? "They've equated [that] a penetration test means, 'I'm secure,'" he said, "What you're doing is hiring someone that tests the defenses that are already in place."
One B-Sides audience member had a more biting reply: "Checkboxers are going to checkbox."
Though seemingly negative on all penetration tests, Hardin actually accepts organizations hiring pen testers to put their current defenses through their paces. But since their value proposition is limited, instead recommended companies take the money they'd pay pen testers and bring a full-time security pro on board.
"If you have no security team and you're paying someone to do a penetration test, just go hire a security person and they will find all that stuff for you," Hardin said. "When should you hire a penetration tester? Basically, when you're out of ideas on how to defend yourself; when you have no more low-hanging fruit."
View all of our RSA 2013 Conference coverage.