BOSTON – Organizations with deep pockets share something in common with firms running application security programs on a shoestring budget. No matter how much money is available, software remediation
“You can’t just helicopter in a team of application security experts, remediate and whip out fixes and then compile everything,” said Wendy Nather, a senior analyst at The 451 Group's Enterprise Security Practice. “There are so many things that can go wrong in one of the sequences to get something fixed, and much of it is organizational.”
Nather, who managed the IT security program at the Texas Education Agency, shared her knowledge of overseeing vulnerability remediation at the SOURCE Boston security conference last week. Nather’s session, “Building a Rube Goldberg Application Security Program,” described how what often seems like a simple remediation project can get sidelined by complex organizational issues and engineering processes.
Major organizations driving application security improvements across the industry have gotten caught up in organizational issues. Microsoft, which touts the security development lifecycle (SDL), admitted in its recent annual SDL progress report that it struggled in the early days of its application security program. When the software giant temporarily halted development of the .NET Framework Common Language Runtime in 2002 to shift focus onto writing secure code, it fixed dozens of flaws over a 10-week period. But the company faced “daunting logistical and technology challenges,” applying what it learned to development on Windows Server 2003. The code base was roughly 10 times that of the.NET CLR. “Organizational size and complexity of the Windows Division also dictated an increased reliance on technology, yet process innovations continued to surface,” the company said in its report.
“Things that we worked on over time have been a combination of not only technical process, but also a process of culture change,” said David Ladd of Microsoft’s software security engineering team in a recent interview with SearchSecurity.com. “When you start talking about improving security practices, it has a large effect across organizations and the technical stuff is usually easier to deal with. The culture change issues are somewhat more challenging.”
In her presentation, Nather described a public-sector project that included dozens of code fixes to an application that was at least a decade old. The single sign-on application was written in ASP and .NET and was “very centrally important to the environment,” she said. “The problem with trying to fix a lynchpin is that it’s a lynchpin; there are so many other apps that are dependent on it.”
While the tactical people turned around fixes in six weeks, the project initially ground to a halt when insufficient funding failed to get a developer with strong skills. In addition, the organization’s QA team had a lot of structure in place. Each one of the dozens of coding errors needed to be mapped, documented and given a business reason. Months went by while “we argued with the QA team a lot on fixes,” Nather said. “They wanted a traceability index for every single instance of cross-site scripting.”
The QA team had to do a build of the code before it could be approved and pushed out into production. Ultimately, the team approved a streamlined traceability process, but the fixes took over a year to deploy, Nather said.
According to Nather, much more training is needed to teach CISOs and other mangers project management and business process engineering. The skills are invaluable when setting out to improve software security, she said.
“It takes a lot of engineering to set something up so it will actually run the way you had planned it,” Nather said. “You have to understand environment and get all pieces in the right order. It’s just as much process engineering as it is the technology side. And this is the part that we don’t teach anybody yet.”