This content is part of the Essential Guide: How to hone an effective vulnerability management program
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

How to pen test: Why you need an internal security pen testing program

Learn pen testing best practices and how to build an internal pen testing team.

In today's complex security landscape, new threats are emerging on a regular basis, and we have more vulnerabilities...

than ever before. As part of a sound security program, most mature security teams have developed a vulnerability management program that includes network and application scanning, patching, and risk assessment. However, many organizations are now asking themselves if it's time to take these programs to the next level by adding penetration testing capabilities into the mix. For many reasons, ranging from compliance mandates to improved vulnerability and threat intelligence, the answer should be a resounding "yes."

Yet there's often some confusion on how best to approach pen testing, what kinds of skills are needed, the tools to use, how often to do it, and what the process should look like in general. We'll clarify best practices for security pen testing and explain how to build an internal testing program and measure its success.

Why you need an internal pen testing team

There are many reasons why organizations should seriously consider performing penetration tests with an internal team. While most organizations today have some type of penetration test performed annually (usually for compliance reasons), penetration tests should ideally be done more often. In essence, a penetration test is a highly specialized, security-specific validation of controls in place. This can range from testing network and application access controls, to software code and IT operational processes.

When developers test code as part of a typical quality assurance (QA) cycle, the testing is done for each iteration of code production to catch bugs. Penetration testing is really a form of QA that looks for flaws in network architecture and design, operating system and application configuration, application design, and even human behavior as it relates to security policies and procedures. If you only performed QA once a year on your applications, would that be enough?

The goal of penetration testing is to find vulnerabilities or flaws in networks, applications, and operating platforms that could potentially be exploited by attackers or simply cause business disruption of some sort. Due to the rapid pace of change in most enterprise IT environments, as well as the rising complexity of most infrastructure, the likelihood of configuration issues and less-than-adequate security controls being implemented increases significantly. Regular penetration testing can be a useful way to determine with a higher degree of certainty that flaws really do exist. However, in order to effectively find these issues before attackers, the testing regimen you put together needs to be focused on consistent, repeatable testing.

In addition to being a security best practice overall, many compliance mandates and industry regulatory guidance specify penetration testing as a requirement or recommendation. There are numerous examples to choose from, some of which are more concrete and others that are more implied. Likely the most well-known standard that explicitly requires penetration testing is the Payment Card Industry Data Security Standard (PCI DSS), which requires pen testing at least once a year "and after any signification infrastructure or application upgrade or modification."

In addition, the PCI Council has released a separate information supplement for penetration testing that specifies who can perform them, as well as suggestions on a methodology and other useful guidelines. The supplement clarifies that internal teams can perform their own penetration tests to meet compliance, as long as they are experienced and are operationally distinct from the areas being tested.

While not explicitly requiring pen tests, HIPAA requires thorough assessments of risks and vulnerabilities to electronic protected health information. According to general security best practices, this should include a technical assessment like pen tests. Pen testing is also included in the Consensus Audit Guidelines (20 Critical Controls) from the SANS Institute.

The pen testing process

The first thing to understand is there are many different types of penetration testing your team may need to perform. Network and system-focused pen testing is still the most common, with an emphasis on traditional server technology, operating systems like Microsoft Windows and Linux, and network-based access and security controls like firewalls, VPNs, routers, and intrusion detection and prevention systems (IDS/IPS). However, more and more organizations are testing Web and other applications regularly, too, and this coincides with the threat landscape we face today. An examination of some of the biggest breaches of 2011 by Chris Wysopal, co-founder and CTO at Veracode, found that two-thirds  were caused by some sort of application issue. To that end, most mature organizations will need to diligently perform both network-focused and application-focused penetration tests.

Most penetration testing regimens follow this cycle, or one similar to it:

  1. Reconnaissance: The "recon" phase of a pen test is focused on information gathering, often through online and openly available sources. Targeted Google queries, DNS and WHOIS lookups, and website searches can provide a vast amount of valuable information about an organization that will come in handy in later phases of the test. Common information gathered includes names and usernames, email addresses, DNS records, keywords that may be included in passwords, evidence of Web-based configuration issues, and sensitive data that has been inadvertently published. Although some organizations skip this phase, those that choose to regularly perform "recon" generally know much more about their organizations' online presence and potential exposure areas.
  2. Scanning and enumeration: Scanning may include website crawling (looking for directory structures and site structure), network scanning with tools like Nmap (looking for active IP addresses and TCP/IP ports that are open), and more in-depth vulnerability scanning that actively probes for specific evidence of existing vulnerabilities in both Web applications and network and operating system components.
  3. Exploitation: Once vulnerabilities have been identified, a pen tester will actively exploit them to try and gain access to systems and applications. Remote server-side exploits are still the most common, followed by Web application exploitation using common techniques like SQL injection, and will also likely include authentication attacks like brute-force password guessing.
  4. Pivoting: After gaining access to systems and/or data, most pen tests should allow for "pivoting," which simply means establishing a new source of attack on the newly compromised target and continuing the scanning and enumeration cycle from that vantage point.

There are quite a few other types of activities that may occur during both network and application pen tests. For instance, if a pen tester gains access to encrypted password hashes, she might choose to start cracking those passwords in the background while continuing with other tasks (if this is in scope for the test). For application tests, client-side code may be enumerated and exploited, possibly allowing for authentication bypass to sites. Some organizations are starting to employ a technique called "fuzzing" to send unusual data into applications to see if error conditions can be caused, thus exposing potential weak points. Advanced security teams may even develop their own exploits. Another type of pen test is social engineering, where the team tries to exploit insecure employee behavior.

How to build an internal pen testing program

To build a pen testing program, the simplest way to get started is by leveraging any existing vulnerability scanning processes and tools you have in-house today. Then, you'll need a skilled team. The skills your team will need vary widely depending on the infrastructure you have in place. Very few organizations perform pen tests against their mainframes, and this is a rare skill for pen testers (although not unheard of). In addition to creativity and curiosity -- which are hard to measure but critical for successful testing -- the following are some general skills that can apply to most pen testing team members, although specialization in networking or applications is common:

  • 3-5 years of operating system knowledge with Linux and Windows.
  • 2-5 years of networking knowledge and skills.
  • Basic scripting/coding skills are critical for pen testers. Real coding experience in C++ and Java are often invaluable, and basic shell scripting is very useful, too. The most common languages used for penetration testing scripting today are Perl, Python, and Ruby.
  • Application development and assessment skills are a must for internal platforms like .NET, PHP, etc. The deeper understanding the team has of the technologies in use, the more effective and efficient application pen tests and code analysis will be.
  • Database skills, with a base understanding of SQL syntax, are becoming increasingly important.

This is not an exhaustive list, of course. Many organizations will have very specific platforms and apps that require specialized knowledge. For example, energy utilities will need testers that understand SCADA platforms and applications and how to test them. There are many sources of good training for penetration testing teams. The SANS Institute has a Penetration Testing course and an Advanced Penetration Testing course that both cover a broad range of topics, and also have specific classes on hacking Web applications, wireless, mobile, and other technologies. Offensive Security, a training and pen test services firm, offers several courses that teach students how to perform penetration tests with the popular BackTrack toolkit, exploit wireless networks, and create exploits, among others.

There are many tools to choose from for performing penetration testing. For enterprises, a number of commercial options are available, including testing suites from Core Security, Rapid7, and SAINT. Open source tools abound, and an excellent starting point for most teams is the aforementioned BackTrack distribution, which is packed with tools that perform scanning, Web application testing, wireless assessment, exploitation, and password attacks. Many specific tools are also included for assessing VoIP and network devices. An excellent Web application-specific toolkit is the Samurai distribution led by SANS instructor Kevin Johnson.

With any of these tools, the key for internal teams coming up to speed is to practice, and then practice some more. The easiest way to accomplish this is with a test lab, preferably containing actual platforms and applications found in the environment. You should also include a variety of vulnerable testing platforms such as Rapid7's Metasploitable, UltimateLAMP, LAMPSecurity, Damn Vulnerable Web Application (DVWA), OWASP WebGoat, and Mutillidae from Adrian Crenshaw. For testing mobile applications, developers can use OWASP iGoat and GoatDroid and the ExploitMe Mobile Labs for iPhone and Android.

Third-party pen testing

Many organizations have a legal or compliance requirement to have an external party perform at least one penetration test per calendar year. In addition to this, it's a good idea to have external firms perform some tests that require extensive knowledge on platforms that your team may not know well, or tests your team is not capable of performing for some other reasons.

It's also a very good idea to rotate through several pen testing firms for a variety of reasons. First, you can ensure the firm you are using does not become too comfortable with your environment and its details; performing regular testing could lead to a scenario where the testing firm becomes complacent and misses potential vulnerabilities. Second, you can get a different perspective from a variety of pen testers, each of whom brings his or her own skills and approach to the table. In general, the more eyes you can get on your environment, the more potential security issues you can find.

Other than these specific cases, performing your own testing is the best way to proceed for all other scenarios. You will be able to test your own environment more often, learn more about the infrastructure, and develop consistent processes for working with IT operations teams to remediate issues. Professional pen testing software companies are creating internal testing tools to allow for team collaboration and consolidated analysis and reporting, as well. Examples include Core INSIGHT from Core Security and Metasploit Pro from Rapid 7.

Measuring pen testing efforts

The key to a successful penetration testing program is to mimic the actual threats you face, emulating attacker behavior and approaches. While this will give you a more reasonable idea of how you may be vulnerable, the key to proving the value is demonstrating the actual impact to business systems and assets by successfully accessing and exploiting sensitive data. Just telling management that you have successfully compromised systems means very little; showing executives how you have accessed payroll or HR data, payment card information, or intellectual property has a much more definitive impact and drives home the severity of issues in the environment.

What kinds of metrics make sense for penetration testing and vulnerability assessments? For vulnerability assessments, common measurements to track include:

  1. Number of vulnerabilities found;
  2. Criticality and types of vulnerabilities;
  3. Percentage of systems and applications scanned;
  4. Number of "unowned" or questionable assets detected.

For penetration tests, the key is a baseline:

  1. How many critical vulnerabilities were found vs. the last test?
  2. User accounts and/or passwords compromised;
  3. Data records accessed.

A pen test should demonstrate real value, both by helping discover vulnerabilities, and potentially making other aspects of the vulnerability management cycle more productive and effective. For example, pen tests can help to cut down on false positives found with vulnerability scanners. Pen tests can also be used as an additional measure to validate application development practices.

Overall, building an internal pen test team is a good idea for many reasons, ranging from false positive validation to compliance requirements. By developing test scenarios that mimic real world threats, and demonstrating true business impact from exploited systems and applications, pen testing teams can help improve the security of your organization dramatically. Most security teams have the skills to get started now and plenty of training is available to fill in the gaps.

About the author:
Dave Shackleford is owner and principal consultant at Voodoo Security, senior vice president of research and CTO at IANS, and a SANS analyst, instructor, and course author. He is a VMware vExpert and has extensive experience designing and configuring secure virtualized infrastructures. He co-authored the first published course on virtualization security for the SANS Institute, serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance. Send comments on this article to [email protected].

Dig Deeper on Risk assessments, metrics and frameworks