alphaspirit - Fotolia
In 2011, Intuit Inc. embarked on an effort to completely revamp its QuickBooks Online accounting service for small businesses with better integration of payroll, customer relationship management and third-party services. With its shift toward software as a service, Intuit decided to transform its desktop production model -- development, QA and IT operations -- so developers could quickly prototype incremental functionality to address customers' demands. As part of the effort, which culminated in the announcement of a completely new service in September 2013, the financial software company focused on responsive development and continuous rollout. In other words, they needed DevOps.
Consumer awareness of massive data breaches started around the same time period. Online attackers scooped up at least 60 million email addresses belonging to Fortune 100 clients of marketing firm Epsilon Data Management in 2011. The same year, hackers caused disruptions in Sony's PlayStation Network for more than three weeks and made off with credentials and information for 77 million PSN and Qriocity accounts. And the colossal Target breach, in which attackers stole 110 million records, capped off 2013. With the introduction of CRM features and payroll information, it's little wonder, then, that Intuit put a major focus on DevOps security.
There's no recipe book for integrating security into DevOps' continuous integration and continuous delivery (CI/CD) cycle, says Shannon Lietz, senior manager for cloud security engineering at Intuit. With 3,000 developers, Intuit faced enormous problems because the security and agile DevOps teams continued to lead separate existences.
"We realized that the DevOps teams were throwing [responsibility] over the wall to security, and we [security] had all the information; we knew all the attacks that were coming in, and the DevOps people…did not have the information to make the decisions," she says. "And we said, 'Alright, we have to figure out how to fix this problem.' It was a head-banging experience."
Fast-tracking DevOps security
Lietz is not alone. Other major firms that created cloud services, or moved their services to the cloud, dealt with similar issues. Video-streaming service Netflix and Etsy, a marketplace for crafters, are well-known proponents of DevOps security. Both companies also pioneered many of the efforts to secure Agile software development.
"There were a million lessons to be learned along the way and some pretty spectacular failures," says Zane Lackey, a former director of security engineering for Etsy, who left and founded Signal Sciences in 2014 with Etsy colleagues Andrew Peterson and Nick Galbreath.
With the online craft marketplace pushing out 50 deploys in a day, Etsy faced several challenges in fast-tracking DevOps security testing and providing visibility to developers.
"How do we get coverage and visibility over the applications and APIs that we are trying to defend?" Lackey says. "And how can we couple that up with the ability to really block attacks against apps that can really scale and be used in production?"
Developers have always pursued faster development methods. With the publication of the Manifesto for Agile Software Development in 2001, and the move from software to cloud services, the practice of developers working more closely with IT operations to automate more of their workflow has taken off.
"DevOps is different," says Rich Mogull, CEO of security consultancy Securosis. "It is as much of a philosophy as an approach. DevOps is about extending the concepts of an assembly line to figure how we get code into the hands of customers."
The DevOps concept -- popularized in The Phoenix Project (IT Revolution Press, 2013), a novel authored by DevOps proponents Gene Kim, George Spafford and Kevin Behr -- has gone from niche to mainstream in the past decade. According to RightScale's "2016 State of the Cloud Report: DevOps Trends," 74% of the 1,060 IT professionals surveyed worldwide indicated their organizations are adopting DevOps for software development, up from 66% in 2015. However, it is not companywide -- 60% of DevOps adoption is by specific business units or project teams.
The fundamental problem? While agile DevOps pushes small batches of code out in days -- and in some cases hours -- security teams work on projects that take weeks and months. To make DevOps more secure, security teams need to have ongoing collaboration with DevOps teams, providing coders with visibility and feedback and automating security checks as early in the development process as possible.
Can security be automated into a set of actions like DevOps? Whatever you call it -- DevOps security, DevOpsSec or Rugged DevOps, which relies on peer-pressure tactics to make developers feel good or bad about their code -- here are four ways to tightly integrate security throughout the CI/CD cycle:
1. Don't be a gatekeeper
When Lackey joined Etsy in 2011, the company already had a strong focus on the continuous deployment model, in which any code that passed the automated testing phase was released into production. The question was how to create a security organization that could really enable the company to quickly push out product features, but in a secure way.
"For security, that feels very scary at first because we have always operated as a gatekeeper," Lackey says. "But the problem is that gatekeeper mentality means, in this modern world, becoming a blocker. Security cannot be the blocker that slows down the business because the developers will route around you."
Training security to not say "no" all the time is only part of the answer. Historically, security has focused on longer projects -- static analysis that required days to run, analyze and build a remediation list -- and that slow feedback loop blocks development in its own ways.
"In the old SDLC [secure development lifecycle] model, we built a lot of security controls that were very heavyweight and very top-down," Lackey says. "And they required a lot of investment to tune and work correctly; and in the end, we would get a report once a month or once a week."
Getting beyond that requires a commitment from the security team to work on DevOps security with developers. Etsy, for example, reused the antivirus toolkit in its continuous integration environment to test source code.
2. Give the developer visibility
Developers want to quickly push out code. When security teams return scan results that show flaws from code checked weeks ago, it requires developers to slow down or, often, to ignore the results.
Instead, security teams should give developers quick visibility into the vulnerabilities in their code. Such visibility helps to tighten the feedback loop, says Joshua Corman, former CTO at Sonatype and co-founder of Rugged Software. (Corman is now the director of the Cyber Statecraft Initiative for the Atlantic Council, which works on international conflict and cooperation.)
Twitter, for example, created a tool called Brakeman to quickly scan submitted code for vulnerabilities and immediately notify the developers. A report, originally in email, lets the developers know when they checked in vulnerable code and how to fix it.
"They found that tightening the feedback loop helped the developers not hate security; it was integrated into their development process," Corman says. "One of the big problems with application security is that there is a time lag between when a bug is inserted and when it is detected."
Providing quick feedback at code check-in is extremely important. Intuit, for example, uses a golden master for its Amazon Machine Image (AMI) infrastructure, and whenever a developer pushes code and creates a new image, their scanners return a letter grade for the security of the image.
At Etsy, Lackey and his team built tools that would deliver security results directly to the developer's queue of defects. That way, security was not a separate process, but part of the development cycle (See "The tools they use").
"We can't just say that security is everyone's responsibility and then not give them visibility or tools for security," he says. "The way in which you actually deliver on that as a security team is by bringing visibility to everyone."
3. Automate security checks and run them all the time
Developing as part of the DevOps CI/CD model means that security has to be as close to continuous as possible.
Netflix, one of the pioneers of CI/CD development, created a collection of programs that would frequently test its systems to make sure they had no vulnerabilities, were configured correctly and complied with company policy. Each program was called a monkey, and the automation suite became Netflix's Simian Army.
Using monkeys -- or automation in general -- a company can assess that accounts are configured correctly, that the most secure and up-to-date version of third-party software is used and that new code is detected, analyzed and vetted through an automated red team process, says Adrian Cockcroft, former cloud architect and director of web engineering at Netflix.
"You need tooling that will break a build if the developer is violating any of these policies," he says. "Once you have built something that you think is good, then it's posted to the testing environment and is automatically checked."
While basing your infrastructure in the cloud is not mandatory, the service-oriented nature of the cloud -- with its APIs -- makes accessing the code and infrastructure much easier for security teams. A significant part of securing DevOps is using that infrastructure to allow programs to automatically check code and the infrastructure.
The service-oriented architecture of the cloud means that applications tend to be less monolithic and, as a result, have smaller codebases. Programs with a million lines of code are not as common anymore.
"Because the size of the codebase has been considerably reduced, the number of bugs has been reduced as well," says Meera Subbarao, director of the secure development practice at Cigital. "However, security has not changed. We still see the bugs in these programs with 200,000 lines of code."
Cigital in May released The Agile Security Manifesto, which suggests four principles to build security into the Agile process. BSIMM 6 -- the Building Security In Maturity Model co-authored by the company's CTO Gary McGraw, a former columnist for this magazine -- was released in October.
4. Don't punish mistakes or blame the developer
When developers move quickly, mistakes are made. And often the security team is the group that finds the errors.
Blaming developers results in negative feedback and worsens relations between developers and security teams, says Lackey. Moreover, developers are not the only ones who will be making mistakes. DevOps security is such a new discipline that most security teams are learning by doing, he adds.
"I certainly made plenty of mistakes along the way," Lackey says. "There were not a lot of other people doing it at the time, but that is how we learned to experiment."
In the end, security groups need to remember that any strategy to secure the agile DevOps cycle that attempts to change how developers work or interferes with their goals will likely fail, says Securosis' Mogull.
"Security needs to learn the developers' language and understand their requirements," he says. "If they do not ship on time, they are in trouble; and for them, that's what matters."
About the author:
Robert Lemos is an award-winning technology journalist who has reported on computer security and cybercrime for 20 years. He currently writes for several publications focused on information security issues.
Find out more about DevOps security tools
Gartner: Why you should add security professionals to DevOps teams
Are DevOps and security a perfect match?
- Windows Server 2012 Security from End to Edge and Beyond –ComputerWeekly.com