alex_aldo - Fotolia
Development teams are continuously under the gun to deliver software. That pressure can make it challenging to maintain code resiliency in enterprise systems or to ensure adequate testing and vulnerability management. According to proponents of Agile development, an approach that encourages teams to demonstrate subsets of functionality early and often to better meet changing business requirements, multiple iterations can lead to higher quality software. But where does that leave security?
“Making the software development process more efficient is an objective shared by Agile developers and security professionals,” says Jim Routh, the CSO for Aetna Inc., a health insurer in Hartford, Conn.
Routh, who not too long ago served as the global head of application security at JPMorgan Chase, has focused on preventing software vulnerabilities and defects for years: He has headed up five software security programs. He is also at the forefront of cybersecurity information sharing initiatives to help industries collectively counter threats; Routh currently serves as the chairman of the NH-ISAC and as a board member for the FS-ISAC. Marcus Ranum caught up with him recently to talk about effective software security programs and the challenges in Agile development and the DevOps process.
Marcus Ranum: I had a rather odd conversation the other day. Someone asked me, 'How do you do software security if you're doing Agile?' And, I immediately wished you were part of that conversation!
I remember we were on a panel a few years ago 'debating' the software security versus Agile question. At the time, you were a proponent of Agile. Are you still? I know you're also serious about application security. How do you bring those two worlds together?
Jim Routh: Agile development and software security are not really two worlds apart. Here is why: Software development is based on iterations to refine functionality and reduce the number of defects in the application, product or utility regardless of the size, scope or complexity of a given system. Agile development is an approach to demonstrate subsets of functionality early in the lifecycle that leads to refinement through many iterations and, ultimately, higher quality by gathering business feedback early, and more often.
Security controls represent tools and techniques to improve software resiliency (withstanding attack by hackers and criminals), which ultimately improves software quality. Making the software development process more efficient is an objective shared by Agile developers and security professionals.
In this case, the definition of efficient is the reduction in the number of flaws and defects through either preventative or detective techniques. A preventative technique (using a template, existing open source library within a repository, or code base previously tested and certified) eliminates the need for labor applied to fixing a flaw or defect, reducing cost in development or remediation post-implementation. A detective technique is simply discovering and fixing a flaw or defect earlier in the lifecycle, resulting in labor being applied to the fix earlier when it is easier and also lower cost. Both sets of techniques (preventative and detective) are used in Agile development and in software security programs, so there is no reason that these belong in two different worlds.
Jim RouthCSO, Aetna Inc.
Applying software security controls to Agile development processes has a higher degree of difficulty compared to a sequential or 'Waterfall' software development lifecycle, and the specific techniques may need some modification. I worked for a large bank that implemented a robust static analysis capability similar to a 'factory' model, where development teams shipped code to an internal utility for static scanning, turning results around in a few days for development teams to consume and make adjustments. This technique and control offered a capability to scan code for defects, enabling the utility to reduce false positives and prioritize the remediation for the development teams. But the teams then had to map the results to changes made in a sprint that took place after the code was submitted to the utility. Agile development teams were burdened with the need for version control to deal with the one- to three-day turnaround of the utility, since sprints were consistently being worked on by developers.
An alternative approach is to use static analysis as a plug-in to an IDE [integrated development environment], enabling developers to scan code whenever they want and get immediate results from the scans. The alternative approach does not find all of the vulnerabilities that a utility does, nor are the false positives identified. But it does fit into an Agile development approach much more effectively and, when combined with other controls, will yield more resilient code.
The growing popularity of a DevOps model for development is another example of improvements to software development that can and must use software security controls, but the techniques for imbedding controls into a DevOps model require adjustment. I believe DevOps actually improves the ability of security professionals to implement effective security controls into the development process, but the control design activity requires some work effort.
Ranum: Can you give an example of how this process might work?
Routh: Here is one example using the same static analysis control I referred to earlier. When a development organization embraces a DevOps model for software development, by using an open source or proprietary framework to provide virtual containers for developers that will run in any Linux environment -- and a base set of operational and infrastructure requirements that are part of those containers -- then it is actually easier to apply security gates to ensure that effective controls have been applied by enforcing a 'signing' of the container before releasing it to QA and production. The container signature and the container metadata provides the artifacts to demonstrate due diligence in the application of the controls, and the signature confirms the authorization for advancing to the next stage.
DevOps applications have a lower cost of support -- in theory -- which means fewer flaws and defects in production. This results in a lower total cost of ownership. The applications (containers) give independence to the developer or a development team to focus on development without worrying about infrastructure requirements, and the base container capabilities ensure the applications will run effectively.
The introduction of security controls -- applied in the right way -- in any development process improves productivity. I have seen productivity gains of 15% and greater when preventative and detective security controls are embedded in the development process, Agile or Waterfall. Productivity gains will increase with the implementation of DevOps as the total cost of ownership will be reduced through the increase in availability and lower costs for configuration and patch management.
Ranum: It's always seemed to me that having a system architect who 'gets' security is one of the victory conditions for any software security program. And I'm guessing that probably goes double with Agile. Am I right? Where does security fit into the architectural process? Is it rolled into the 'inspect and adapt' part, or do you make it the responsibility of the Scrum master?
Routh: It is not only a 'victory condition,' it is a rarity. Teaching architects at scale how to use techniques like threat modeling to understand security requirements in application design is difficult under the best of circumstances. I have used many different approaches in the past with generally poor results.
I've observed some recent success with training architects to understand threat modeling done in design and injected into the review of business requirements that lead to sprints and trains. The architects are not Scrum masters and act more like advisors.
We prefer to encourage Scrum masters to be the process owners who understand what techniques and controls are required in the sprints. There are a few exceptions -- when Scrum masters have a security background and appetite -- where they will influence the design work within development teams, but these are rare. The architects have their own curriculum for software security, and they influence development teams while also recommending design considerations for security.
The recent success of teaching architects is largely due to some innovation of my program leads and the top-down leadership of the chief architect enforcing the implementation of the software security curriculum for all architects. The general awareness of the need for security is no longer a debate; it is simply a reality that every architect understands. The control design that has been applied to the overall development process is both understood and supported to the point where Aetna has many architects that understand information security requirements and influence development teams as needed.
Ranum: Do you do any specific security-related training for your Scrum masters? And, if so, what is it?
Routh: Yes we have mandatory software security e-learning classes assigned to Scrum masters in the Enterprise Learning Management system that cover software security fundamentals and defensive programming. There are additional classes available that are optional and recommended for the Scrum masters that cover application architecture and system testing. Scrum master is one of 16 different curriculums assigned to specific roles in the development process, with assigned and optional classes that are specific to security.
Ranum: I have to say I prefer the DevOps model to Agile. But I see both of them as a reaction to the Waterfall model. Anyhow, I love the fact that you can talk quantifiably about productivity gains in your development process. I'm fascinated by metrics, and I'm sure you must be instrumenting your process fairly effectively. Can you tell us a bit about how you measure your process?
Routh: Sure, there are two types of measures that we apply to determining productivity of development with the security controls imbedded within the process:
- Preventative controls that eliminate the need to correct defects and flaws; and
- Detective controls that identify defects early in the lifecycle, so the defects are fixed in development where the cost of fixing it is the lowest.
Both control types are measured based on a standard cost for development time multiplied by the amount of labor effort that is offset by the controls. For example, when we use an input/output validation standard development framework -- we use a version of Enterprise Security API -- the amount of labor to fix code vulnerabilities (that are prevented) is measured resulting in a productivity gain for the developer (no defect-remediation work) and their time can be reinvested in developing code.
In another example, a static analysis tool is used by the developer whenever they wish to scan code for defects. The developer must demonstrate the code they deliver into a build meets a specific threshold for security. The implication is that they fix the defects in order to pass the security threshold, and so we count the defects identified (high risk only) and then calculate the delta in development cost to fix a standard defect during development vs. post production. The result is an increase in productivity. The savings in productivity is reinvested in developing more functionality, so the actual development dollars are not reduced but the productivity of the developers' time improves. These two measures determine the overall productivity improvement that justifies the up-front cost to implement the controls. They are KPIs [key performance indicators] for the development leads and are reported at all levels in the application portfolio (development leads, VPs, CIO).
Ranum: So if you were talking to an exec who was about to inherit a development process that included Agile or DevOps, and you could only give them one suggestion, what would it be?
Routh: Let the economic benefits drive adoption and implementation and don't lead with the risk benefits. The productivity gain in development and the operating cost savings for a lower total cost of ownership (DevOps) are easier to measure than risk mitigation or avoiding risk.
About the author
Marcus J. Ranum, the chief of security at Tenable Network Security Inc., is a world-renowned expert on security system design and implementation. He is the inventor of the first commercial bastion host firewall.