While it can be helpful, perhaps in the context of budgeting and office politics, to present your boss with a report that proves even someone who has no inside knowledge of the new website can hack into it, I have several reservations about the zero-knowledge approach. We know that a certain percentage of attacks are going to come from inside the network perimeter, or from the outside with insider help. If you want to know how secure your site is across all real-world scenarios, zero knowledge is not necessarily the best starting point.
A zero-knowledge approach also has the potential drawback of being slower to return results. If you describe some of the basics of your system to the tester beforehand, it can save time, and time is often tight when a new product is being rolled out. One important variable here is the status of the target: is it in production or in development? When testing a production system, you may want testers to let you know about a gaping hole as soon as it is discovered, rather than waiting until the final report. Provided the contract with the tester is appropriately worded, you may be able to patch the hole and get the patch tested. Indeed, some would argue that treating a pen-test as an iterative improvement in security is better bang for the buck.
Finally, whether you choose to proceed from a zero-knowledge starting point, remember that you can't truly replicate the real world without breaking the law. You must assume your attackers are prepared to commit illegal acts to achieve their ends, but few organizations are in a position to give their pen-testers a get-out-of-jail-free card. So, you will want your pen-tester to be able to think like a criminal hacker and document for you those methods of penetrating the system that rely on illegal acts.
The bottom line is that the real world and a pen-test are two different things, and your security money may be best spent having seasoned security experts explore the potential vulnerabilities of your product while armed with plenty of knowledge about it, rather than setting up unrealistic testing scenarios.
- Get advice on how to use penetration testing to help with compliance efforts.
- Social engineering testing and penetration testing: Learn how they mix.
This was first published in May 2008