A new report has sparked a war of words among vendors, kindling a debate about the best way to test information security products and the fairness of evaluating a product without its manufacturer's cooperation.
We do not take a single penny. Any time we do a public test, any content that is published, there is no money that changes hands.
CEO, NSS Labs
Independent infosec research and advisory firm NSS Labs Inc. recently tested six of the leading enterprise breach-detection systems, releasing the findings last week in its report, Breach Detection Systems Comparative Analysis and Security Value Map.
In the report, NSS Labs gave positive ratings to products from Cisco-Sourcefire, Trend Micro, Fidelis and Fortinet, but offerings from FireEye and AhnLab were placed in the "caution" category, meaning those products offer "limited value for money given the three-Year TCO [total cost of ownership] and measured security effectiveness rating."
Dave Merkel, chief technology officer for Milpitas, Calif.-based threat-detection vendor FireEye Inc., immediately lambasted those results, noting that the company's product discovered 11 of 13 zero-day vulnerabilities that were known to have been exploited in 2013.
"Any lab test is fundamentally unable to replicate the targeted, advanced attacks launched by sophisticated criminal networks and nation-states," said Merkel in a statement to the press responding to the report.
In a blog post last week, Manish Gupta, senior vice president of products for FireEye, provided a more in-depth response to the NSS Labs report, stating that the company participated in a previous NSS testing process in 2013, but after criticizing "flaws of the testing methodology," dropped out of the most recent round of tests. Gupta claimed that FireEye's product detected 201 of the 348 malware samples thrown at it in the first test, with FireEye considering 177 of the missed samples duplicates and a further 19 as corrupted. He described the final 11 samples as "non-malicious".
Gupta specifically criticized the company's reliance on known malware samples, most of which he claims were taken from the free malware-scanning service VirusTotal, while not including any of the zero-day attacks that FireEye's products are known for spotting. The researchers at NSS Labs also didn't have access to FireEye's Dynamic Threat Intelligence cloud service, according to Gupta, rendering the product less effective than normal.
"We respect NSS and the work they do -- especially for IPS -- and their testing methodology for BDS is also more suited to testing IPS products," said Gupta. "However, we believe the issues we identified with their evaluation of advanced threats are indicative of the security industry's broader lack of knowledge regarding sophisticated attacks."
NSS Labs: A 'pay-to-play' model?
While FireEye's criticisms largely focused on the testing methods in place at NSS Labs, Nir Zuk, chief technology officer for network security vendor Palo Alto Networks, alleged that the research firm depends on a pay-to-play model to make money off such reports.
Speaking to CRN.com, Zuk said that NSS Labs issues "reprint rights" to vendors featured in its reports to publish the test results, with the negotiated fees for such licenses sometimes exceeding $100,000 if the tests aren't yet public. This practice, he said, enables "mediocre vendors" to be more competitive.
"You don't want to do a report on two vendors; you want to do a report on 10 vendors and charge each of them $100,000 -- and that's how you make money," Zuk told CRN.com. "If you are going to have a very high bar for your test and only one or two vendors are going to succeed in your test, the vendors are going to stop paying for it."
In response to a SearchSecurity inquiry, a FireEye spokesperson said that Gupta's blog post "captures our objections to the test fairly extensively" and declined further comment.
Vikram Phatak, CEO for NSS Labs, blasted the allegations made by Zuk, noting that he was compelled to apologize to Phatak's company in the past for making similar remarks.
"We do not take a single penny. Any time we do a public test, any content that is published, there is no money that changes hands," said Phatak. "Now whoever the top two or three vendors are out of however many, likely because of the credibility of our tests, are going to purchase the reprint rights to promote the content, but that's the same thing as if somebody does well in Consumer Reports or if somebody does well in Gartner's Magic Quadrant."
As for FireEye's criticisms, Phatak emphasized that the company was "fully aware" that its product was being tested and that its engineers were actually involved in the on-site installation process at the research firm's facilities. Furthermore, he said the full forensic information produced by the tests was provided to all of the vendors in the report, including FireEye, and all were offered ample opportunity to comment prior to the report's release if they thought something had gone awry.
Though he noted Gupta's criticism of the malware sample, Phatak said that the other vendors included in the report did not raise the same issues.
An 'agnostic' approach to security product testing
In its report, NSS rated breach-detection systems based entirely on their ability to detect attacks that slipped by other security technologies like antivirus or an intrusion prevention system, according to Phatak. He continued that vendors were given credit if they detected the attack at any of what he considered five stages in the exploitation process, including the initial infection and the malware's callback to its command-and-control infrastructure.
Phatak said the testing process was based on the requests of NSS Labs' enterprise clients, who simply wanted better awareness of potential breaches in their environments.
"In fact, if there was a sixth way that somebody came up with that we weren't aware of and it detected something, that's fine," said Phatak. "If it shows up in the logs that it detected the infection, that's all we care about. It was very much an agnostic approach."
Al Huger, senior director of engineering for Cisco Systems Inc.'s security business unit, whose Sourcefire AMP product was part of the NSS Labs report, described the testing process as "comprehensive," especially in comparison to other private testing firms that often give vendors the opportunity to improve results before they are finalized. He said that NSS Labs works with vendors to ensure the initial setup of a product is correct, but after that, it's very much a "hands-off" affair.
The malware samples used by the NSS Labs researchers were also completely valid for this sort of evaluation, according to Huger, as they were taken from attack campaigns that were live on the Web at the time of testing. The tests went beyond mimicking real-world scenarios, he said, apart from the atypical volume of malware that was thrown at the products.
"It was the full threat lifecycle, from exploitation over the wire onto the endpoint back out again onto the Internet," said Huger. "[It was] identical to what you see in enterprises today, just the volume [of malware] was different for the test."
Ultimately, Phatak said that FireEye's product simply did not perform as well as those from other vendors, but based on the intelligence of the FireEye team, he expects the vendor to spend more on research and development and will come back with a stronger product for future tests.
"Frankly, [FireEye] is a public company," said Phatak. "It's quite shocking to see them say things that are so provably and completely untrue."