New vulnerability reports are released constantly, but information security industry observers say the value provided...
to enterprises by such reports may be increasingly suspect, and that enterprises may be better served focusing less on vulnerabilities and more on risk management.
The debate is highlighted by an ongoing tussle between a security researcher and Copenhagen, Denmark-based security provider Secunia ApS. In its 2015 Vulnerability Report, released last month, Secunia claimed that had cited 15,435 vulnerabilities across 3,870 applications in 2014.
However, questions regarding the methodology used to arrive at those figures and the transparency in how they are presented have sparked controversy.
Brian Martin, content manager of the Open Sourced Vulnerability Database and president of the Richmond, Va.-based Open Security Foundation, said that Secunia both artificially inflated vulnerabilities and underrepresented vulnerability totals for the year, due to what he calls a controversial counting methodology.
Martin believes that the statistics presented by Secunia don't compare easily to other vulnerability databases, which can cause confusion, and Secunia's lack of transparency regarding how its numbers are generated create questions about the conclusions drawn from the data.
Martin noted that no vulnerability database can catalog every vulnerability that is found throughout a year, due to limitations of time and resources, but he suggests certain standards for a minimum baseline.
"Their 15,000 number has no bearing with the rest of the world; it cannot be compared to other vulnerability databases, like MITRE's CVE, NVD, IBM X-Force, or Symantec BID. Secunia does not maintain a 100% mapping to CVE," Martin said. "At the minimum, every vulnerability database should map to CVE, and if they don't, you should ask why."
In a statement, Secunia maintained that it stands behind its methodology, but declined to comment further for this story.
Unending stream of vulnerabilities
However, Martin said that the inability to catalog every vulnerability is not an issue unique to Secunia's research methods, because there are simply too many vulnerabilities found each year. As an example, he mentioned research done late last year by Will Dormann, vulnerabiltity analyst with the CERT division of the Carnegie Mellon University's Software Engineering Institute in Pittsburgh, which found tens of thousands of Android apps with vulnerabilities related to the proper validation of SSL certificates. He noted that these vulnerabilities added up to more than 23,000, but most databases stopped cataloging them after about 1,000 because the apps affected became too obscure.
Dormann said that he was asked to stop assigning CVE numbers to the Android vulnerabilities he found because there were simply too many.
Martin described two general types of vulnerabilities. First are those like the above-mentioned Android app vulnerabilities in which a defined protocol was implemented incorrectly. He suggested these should be counted as separate vulnerabilities in each occurrence because to patch the implementation flaw, each fix will be different on a coding level.
Where Martin said Secunia artificially inflated its numbers was with the second type of vulnerability. He used the example of Heartbleed, which was based on a flaw in OpenSSL, meaning that any software that used OpenSSL would "inherit" the vulnerability. Martin contends that no matter how many different products this type of vulnerability affects, it should count as one vulnerability because the ultimate fix boils down to updating the OpenSSL library within the software, a process that will be the same regardless of the uniqueness of each software product affected.
In its vulnerability report, Secunia counted each occurance of the Heartbleed flaw separately, which Martin said meant that it would be counted hundreds or thousands of times, and could ultimately cause confusion when compared to other databases, or used for analytics.
"I think it's completely wrong to use that methodology if you are going to provide statistics based on that," Martin said of Secunia's practice, and urged that transparency is the best solution. "Explain what the numbers mean; don't call them vulnerabilities, but examples of vulnerabilities. It's just that simple to disclaim what that number really means."
Focus on risk exposure
Dormann said that people often get drawn in to vulnerability counts because they often think that there are conclusions to be drawn from the data, but that can be a pitfall. He said that spikes in vulnerabilities are often caused by new tools that make vulnerabilities easier to find, or what he calls the "children's soccer effect" where researchers cluster around a new product or technology.
"Vulnerability counts are a very narrow and dangerous metric to focus on, and there isn't much benefit to focusing on such counts," Dormann said. "In a lot of cases, there are things that can be done to proactively protect against vulnerabilities. If something isn't exploited today, someone will likely exploit it eventually, so it's good to have protections in place."
Johannes Ullrich, dean of research for the Bethesda, Md.-based SANS Technology Institute, found Secunia's methodology to be sound, because he looked at the issue in terms of risk.
"If you have two or three servers that have the Heartbleed vulnerability, it increases exposure," Ullrich said. "What you really need is to assess your risk. Secunia shows the exposure, but it doesn't talk about the likelihood of attack. It's only half of what you need."
Ullrich suggested that focusing on the vulnerability numbers could distract enterprises from the real work of patching systems. He pointed to the recent study showing that many companies are still at risk to Heartbleed because of incomplete patching efforts.
"Enterprise needs to have a good process for applying patches, make sure they have been patched properly, and keep a log of what has been done and what needs to be done," Ullrich said. "If you just apply the patch, you may not get the other things like configuration changes or certificates."
Martin agreed that the focus needs to be on relevant risks, because he said that the vulnerability numbers are only relevant the moment they are released. He showed how when he first presented the vulnerability count for 2012, the number was just over 8,800, but now the 2012 year contains nearly 10,500 vulnerabilities.
"Why does an organization care how many vulnerabilities were cataloged? It means more to security companies that monitor these things," Martin said. "Enterprise should be more focused on the vendors they use and the vulnerabilities found for each. They should be asking, 'What vulnerabilities affect us?'"
Learn more about software patching myths