rvlsoft - Fotolia
Even the best software has flaws, but with key controls early in the development cycle, security defects can be vetted along with functional testing and quality assurance. While many enterprise security officers fail to find time for application security, James Routh, CISO for health insurer Aetna, is leading his fifth software security program.
Building a software security program can improve software resiliency and developer productivity, according to Routh, who this month discusses with contributor Marcus Ranum what he's learned about implementing money-saving programs from the top-down. He also looks at what's ahead for software security, as mobile applications and open source components broaden the challenge for many enterprises.
Jim, thank you for taking the time to talk about software security programs. This is one topic that I think is incredibly important, which still doesn't get enough attention in the industry. You've been promoting software security programs for a long time. How did you 'get it'? What happened that enabled you to understand the importance of having a software security process?
Jim Routh: My first role in information security over a decade ago was as a CISO for American Express. And I was fortunate to have worked with Michael Barrett (former CISO for PayPal) and Mark Merkow (software security lead for Charles Schwab brokerage services), who told me that the software security program needed a lot of attention, because it was based on a single review process at the end of the life cycle that did little to teach software developers how to adjust their practices to reduce defects.
At the time, most new development was Web applications. I realized that the attack surface was getting larger, and I wanted to avoid security incidents -- this was back in the day when avoiding incidents was even plausible -- so I embarked on educating application developers on secure coding practices.
I think I learned a lot more than they did. The developers and their project leads viewed security controls in the development process as obstacles to getting real work accomplished, and they had no budget for the hours of remediation work required when security vulnerabilities were discovered. My implementation approach was effective at giving me an education but not that effective at educating them.
Mark subsequently figured out how to give them the education they needed after I left and went to The Depository Trust & Clearing Corporation. At DTCC, I realized how vital it was to improve software resiliency to avoid business disruptions and breaches, which could potentially cause a lack of confidence in the financial markets globally. I adjusted my approach and obtained top-down support from the CEO to the CIO to implement a comprehensive software security program based on the simple premise that producing software with fewer defects costs less money to support, and discovering flaws earlier in the life cycle takes less time to fix. I also compared the practices implemented with a body of knowledge that emerged called the Building Security In Maturity Model or BSIMM. It enabled me to compare the maturity of the controls implemented with those of eight other companies, like Microsoft, that invested in software security programs.
That experience helped me understand how to gain support for the program based on the economic benefits while investing in developer education. I was able to measure the decrease in vulnerability density, when developers used static analysis tools in the development process and learned how to avoid defects. I learned that security vulnerabilities discovered in the development process are nothing more or less than defects that need to be prioritized and fixed. Integrating security defects with functional flaws and identifying the high-risk defects made it easier for the development teams to assign responsibility for fixing them.
I still use the economic benefits to convince organizations to implement a mature software security program, and I've learned that application developers today embrace the idea of discovering defects early and improving the resiliency of software. At JPMorgan Chase, I taught software security controls to more than 17,000 developers globally. They were very receptive to learning more effective ways to develop systems. I set up a model in which selected developers got extensive education on software security and they would help other developers apply the techniques to reduce defects. At Aetna we call them security mavens, and they enforce the key controls.
Can you tell us a bit about how you do software development at Aetna?
Routh: Software development has evolved to where most custom application development starts with the consumption of open source, so I've added a control to enforce the use of selected open source frameworks based on the security risk of the libraries from scan results. I also have some new cool technologies for mobile applications that deal with the unique security risks in the mobile application distribution system.
There are always objections to a software security program. I am sure you know them, chapter and verse: We can't slow things down. We can't be constrained. How do you overcome those objections?
James RouthAetna CISO
Routh: It's really hard to argue against a process that saves money, improves quality and reduces risk. I come armed with simple information demonstrating each attribute and a commitment, which is that having done this so often it is relatively easy to address. The developers don't push back today but project managers still do, until they understand the benefits. Many of the tools I use today for mobile applications are not well known, so I typically get push-back from mobile developers who are unfamiliar with them. In those cases, I simply use the industry data available to show how easy it is to compromise a mobile application and release it through secondary channels. Most of the vulnerabilities exploited today are the result of breaks in mobile software distribution.
The bottom line is that selling a software security program is straightforward when you use the financial benefits as the driver. Implementing a successful program is more about changing behavior than implementing technology, so positioning security as an attribute of software quality reaffirms the role of the development lead -- which is to own the process and the results -- while security designs the controls and measures effectiveness primarily for the benefit of the development lead.
One thing that jumps out at me is that you're leveraging metrics. You're not just saying, 'This is the right thing to do.' You're able to explain how it saves money to avoid losses and out-of-cycle maintenance. Can you tell us a bit about your metrics?
Routh: There are two fundamental drivers of productivity improvement once security controls are implemented into the development process: First, frameworks like the Enterprise Security API and tools for open source component selection and static analysis prevent defects from getting into the build process, so this eliminates the cost of fixing flaws and defects. Second, detecting defects in quality assurance, and then fixing the high-priority defects, costs less in terms of time and, therefore, money compared to fixing defects discovered in production.
So I calculate the average time it takes to fix a defect post-production (highly complex defect fix = 8 hours; moderate complexity = 4 hours; simple fix = 2 hours) and compare it to the average time it takes to fix a defect in development, and then multiply this by the number of high-risk defects discovered. I use the standard recovery amount per hour, for example $125, as the cost -- this is an average industry hourly cost of FTE developers along with vendors and third-party consultants doing development. The result is a 'productivity save' for all defects prevented, in addition to the savings of moving defect fixing from post-production to quality assurance or pre-production. The productivity save for development time is reinvested in more functionality in less time with the removal or reduction of the defect-fixing work effort.
What I don't understand is why so few executives seem to 'get it' like you do. Do you have any idea what we can do to broaden the understanding of this topic? I feel like with the last 15 years of 'penetrate and patch' -- and the painful examples of major software vendors constantly pushing bug-fixes -- if that's not a danger sign, what is? What can we do?
Routh: More than 70 companies measure the maturity of their software security programs today in the BSIMM data, so the body of knowledge in software security has grown significantly. Your question about why so few leaders get it is a reasonable one, but my point is that many do understand the economic drivers for software security and have implemented more effective controls across industries. We know a lot more about software security practices today than we did a decade ago, and the information that I've shared with you is available in the public domain.
Many of the software vendors that push bug-fixes also have mature software security programs established -- Microsoft, Adobe Systems and EMC, to name a few. The reality is perfection and software are two words that don't belong in the same sentence, and there will always be opportunities to identify and improve software defects.
We are trying something new by taking a Web application that we've applied all of our controls to and essentially crowd-sourcing the pen testing of the app to see what we learn. This means that dozens of software researchers will perform pen tests and share their results with us.
I fully expect to find new defects from this process despite the dozens of tests performed on the same software version previously. It also is likely to help me understand how the many controls deployed in our development process are performing and where there are opportunities for improvement. We track defect density over time, and the data clearly shows a dramatic improvement from the developers who are using better tools to detect defects in development.
For as long as we build software there will be defects discovered throughout the process and opportunities to improve quality. What is clear is that making investments in improving quality makes good economic sense, and perfection in software will remain elusive for the near future.
About the author:
Marcus J. Ranum, chief security officer of Tenable Security Inc., is a world-renowned expert on security system design and implementation. He is the inventor of the first commercial bastion host firewall.
- Windows Server 2012 Security from End to Edge and Beyond –ComputerWeekly.com
Dig Deeper on Secure software development
Synopsys on source code security sensitivities
Marcus Ranum Q&A: Aetna CSO Jim Routh on Agile security testing and challenges
Commercial software more secure than open source, finds report
Tracking bugs effectively in continuous development