The following is an excerpt from the Official (ISC)2 Guide to the CISSP CBK, fourth edition, edited by Adam Gordon, CISSP-ISSAP, ISSMP, SSCP. This section from Domain 6 provides an introduction to several security testing techniques that can be used in and out of test environments to identify -- and mitigate -- vulnerabilities in software.
In software development, a small coding error can result in a critical vulnerability that ends up compromising the security of an entire system or network. Many times, a security vulnerability is not caused by a single error, however, but rather by a sequence of errors that occur during the course of the development cycle. A coding error is introduced, it goes undetected during the testing phases, and available defense mechanisms do not stop a successful attack.
Security must be a priority in all phases of software development. Effort should be aimed at preventing software vulnerabilities with security testing techniques -- not only detecting any issues before release, of course, but also limiting their practical impact, for example, by reducing the product's attack surface. Most security vulnerabilities are caused by one of the following four reasons:
1. Bad programming patterns, such as missing checks of user-influenced data, e.g., in SQL injections vulnerabilities;
2. Misconfiguration of security infrastructures, e.g., too permissible access control or weak cryptographic configurations;
3. Functional bugs in security infrastructures, e.g., access control enforcement infrastructures that inherently do not restrict system access;
4. Logical flaws in the implemented processes, e.g., resulting in an application allowing customers to order goods without paying.
Comparing common security testing techniques
The vast majority of successful attacks against IT applications do not attack core security primitives such as cryptographic algorithms. Attackers much more often exploit had programming, interface problems, uncontrolled interconnections, or misconfigurations. From a high-level perspective, security testing techniques are often classified as follows:
Black-box testing vs. white-box testing
In black-box testing, the tested system is used as a black box, i.e., no internal details of the system implementation are used. In contrast, white-box testing takes the internal system details (e.g., the source code) into account.
Dynamic testing vs. static testing
Traditionally, testing is understood as a dynamic testing, i.e., the system under test is executed and its behavior is observed. In contrast, static testing techniques analyze a system without executing the system under test.
Manual testing vs. automated testing
In manual testing, the test scenario is guided by a human, while in automated testing the test scenario is executed by a specialized application.
When selecting security testing techniques or tools, the security practitioner needs to consider many different things, such as:
Different security testing methods find different vulnerability types.
Different security testing methods behave differently when applied to different application types.
Quality of results and usability
Security testing techniques and tools differ in usability (e.g., fix recommendations) and quality (e.g., false positives rate).
Security testing tools usually only support a limited number of technologies (e.g., programming languages), and if a tool supports multiple technologies, it does not necessarily support all of them equally well.
Performance and resource utilization
Different security testing techniques and tools require different computing power or different manual efforts.
It is commonly accepted that fixing bugs and security vulnerabilities late in the software development is usually more costly compared to fixing them as early as possible. Therefore, security testing techniques should be applied as early as possible in a secure software development lifecycle and not as an afterthought.
Assessing possible attack surfaces
Strictly speaking, a security review of the architecture and threat modeling are not security testing methods. Still, they are an important prerequisite for subsequent security testing efforts, and the security practitioner should be aware of the options available to them. The selection of security testing techniques is:
Architecture security reviews
A manual review of the product architecture to ensure that it fulfils the necessary security requirements.
Prerequisites: Architectural model.
Benefit: Detecting architectural violations of the security standard
A structured manual analysis of an application specific business case or usage scenario. This analysis is guided by a set of precompiled security threats.
Prerequisites: Business case or usage scenario.
Benefits: identification of threats, their impact, and potential countermeasures that are specific to the development of the software product.
These methods help to identify the attack surface and, thus, the most critical components. This allows a focusing on the security testing activities in order to ensure that they are as effective as possible.
Assessing security outside test environments
In the development stages where an application is not yet sufficiently mature enough to be able to be placed into a test environment, the following security testing techniques are applicable:
Static Source Code Analysis (SAST) and Manual Code Review
Analysis of the application source code for finding vulnerabilities without actually executing the application.
Prerequisites: Application source code.
Benefits: Detection of insecure programming, outdated libraries, and misconfigurations.
Static Binary Code Analysis and Manual Binary Review
Analysis of the compiled application (binary) for finding vulnerabilities without actually executing the application. In general, this is similar to the source code analysis but is not as precise and fix recommendations typically cannot be provided.
Executable in a test environment
At later stages of the development lifecycle, when the software can actually be executed, further security testing techniques can be applied, such as:
Manual or Automated Penetration Testing
Simulates an attacker sending data to the application and observes its behavior.
Benefits: identification of a wide range of vulnerabilities in a deployed application.
Automated vulnerability scanners
Test an application for the use of system components or configurations that are known to be insecure. For this, pre-defined attack patterns are executed, and system fingerprints are analyzed.
Benefits: Detection of well-known vulnerabilities, i.e., detection of outdated frameworks and misconfigurations.
Fuzz testing tools
Send random data, usually in larger chunks than expected by the application, to the input channels of an application to provoke a crashing of the application.
Benefits: Detection of application crashes (e.g., caused by buffer overflows) that might be security critical.
All these security testing techniques require a deployed and configured application on an isolated test system, including back-ends or external services. While dynamic techniques usually do not achieve similar coverage of the analyzed application in relation to static approaches, they are particularly well suited for detecting vulnerabilities that involve data flows across system boundaries. The greater the parity between production and test environments, the higher the assurance of test results from the test environment reflecting what will happen in the environment.
CISSP® is a registered mark of (ISC)².