News Stay informed about the latest enterprise technology news and product updates.

Testing, assessment methods offer third-party software security assurance

No ultimate test can give third-party software a clean bill of health, but careful assessment can help organizations gain more control over vendors.

Gary McGrawSoftware is not created equal, especially when it comes to security. I’ve done my fair share of talking in this column about how to create and measure a software security initiative to make sure the software you build yourself is secure and I’ve even talked about how to get started with a brand new software security initiative. How can you tell whether the software you buy or outsource to others to build is secure enough? Do you trust your vendors? Do all vendors do the same thing when it comes to software security? (Hint: the answers are “good question,” “why?” and “no.”)

Every enterprise depends on software

Every modern enterprise uses lots of software. Some enterprise software is homegrown, but a vast majority of enterprise software is third-party software built and maintained by outside vendors. Third-party software itself comes in several flavors: it can be custom built to specification, it can be commercial off-the-shelf software (COTS), and it can live in the cloud as part of a Software as a Service (SaaS) model.

Many large firms are working hard on vendor control and supply chain security issues. This is especially apparent when it comes to software vendors providing goods to financial services organizations. A typical multi-national bank has thousands of vendors, several hundred of which directly impact software security posture.

Big firms are busy exploring two basic options for software security and vendor control. The first involves directly assessing the security of a particular piece of software. The second involves measuring the software security capabilities of a vendor. Both approaches are valuable.

Third-party software security assurance: Measuring software directly

A badness-ometer approach to software measurement works on the same theory as penetration testing—try breaking something and see how far you get. The idea is to carry out a series of straightforward black box tests against a given application. If the canned tests break the software, you know it’s truly bad and should not be trusted. (On the flipside of the coin, if the canned tests don’t break the software, well, you have a very small amount of evidence that the software is secure.)

The good news about badness-ometers is they are straightforward and cheap to apply, especially when it comes to measuring third-party code. Just aim the tests at the application in question and away you go. If the application fails the tests, then let the vendor know the application they built is not good enough to use. Firms like Veracode and my firm Cigital will even carry out these kinds of tests for you. Largely the direct measurement approach is cheap enough that you can apply it to your entire portfolio.

There are two main drawbacks to direct measurement. The first is that software is always changing, and direct measurement is limited to a somewhat cursory point-in-time look. Think about how often the software you rely on automatically updates itself, and multiply that number times distinct platforms, geographic locations, software environments, and so on. Should you really have to constantly test all of your vendor’s code?

About the [In]security column

This monthly security column by Gary McGraw started life in print in IT Architect and Networkmagazines and was originally called “[In]security.” That was back in October 2004. The column then transitioned into Web content at several publications before finding a home at SearchSecurity. You can always find pointers to the complete [In]security series on McGraw’s writing page. Your feedback on the column is greatly appreciated.

The second main drawback to direct measurement is that badness-ometers are not security meters, no matter how much we would like them to be. There is no ultimate set of tests that can ensure security, so don’t let anybody tell you there is (especially a software vendor). Direct measurement is useful and economically feasible, but it’s no panacea.

Measuring capability with BSIMM and vBSIMM

Microsoft has spent a ton of time and treasure both creating and marketing software security and the secure development lifecycle (SDL). If you are a big firm, you most likely rely on Microsoft software in some capacity. In a way, the existence of the SDL provides some peace of mind that Microsoft is paying attention to security and attempting to build software you can rely on. But what about your other vendors?

Enter the BSIMM, an ever-expanding study of 51 firms’ software security initiatives. All 51 firms participating in BSIMM4 have a software security initiative underway and an SDL equivalent (though they are not all at the same level of maturity). The BSIMM measurement is a way of describing, comparing and contrasting these SDLs. And the BSIMM Community as a collective is very serious about software security.

At the most basic level, participation in the BSIMM may be enough to winnow the grain from the chaff among your software vendors. Sadly, some vendors may not even be able to spell the word security, or may wonder exactly what in the heck you are talking about when you query them about it. Those would be the vendors to worry about. The vendors with an active software security initiative? They’re probably not going to be your biggest risk.

Of course, BSIMM participation is intense. It involves direct in-person measurement of 111 software security activities, deep dives into particulars, and an objective scoring system. As such, participation in the BSIMM may be too high a bar for vendor control.

That’s why we created the vBSIMM with the help of JP Morgan Chase. If you think of the BSIMM as a measuring stick, you can think of the vBSIMM as a ruler. When it comes to setting the right height on the vendor software security bar, the vBSIMM—which measures only 15 software security activities (instead of 111) and relies on attestation—provides a much lighter weight alternative to the BSIMM.

The vBSIMM scheme is far from perfect and it does nothing to guarantee that any particular vendor product is actually secure enough for all uses. But the vBSIMM scheme is far superior to no vendor control at all. It’s particularly useful when a vendor produces multiple applications for you.

There are drawbacks to an approach like the BSIMM/vBSIMM, since the metrics involve an indirect, process-oriented measurement of software security capability. The real questions are, “How well does the software security initiative in question actually work? Are the activities carried out by your vendor effective? Does the vendor really create secure code consistently?”

Vendor control

In the end, an approach that combines both indirect measurement of capability with direct measurement of applications is probably the way to go. At any rate, software security is just as important when it comes to your software vendors as it is when it comes to your own developers.

About the author:
Gary McGraw, Ph.D., is CTO of software security consulting firm Cigital. He is a globally recognized authority on software security and the author of eight best-selling books on this topic.

This was last published in January 2013

Dig Deeper on Secure software development