V. Yakobchuk - Fotolia
If controversy is good for business, business is no doubt booming for NSS Labs Inc.
The Austin, Texas-based security product testing firm's new report on next-generation firewalls has caused quite a stir, specifically in regard to its results for one vendor: Palo Alto Network Inc.'s appliance, which garnered a "recommended" score last year, fell to a "caution" rating this year.
The NSS report, a combination of its Next Generation Firewall Security Map (SVM) and Comparative Analysis Report (CAR), was released Tuesday. Palo Alto, which is generally seen as an industry leader in the next-generation firewall category, was put to the test along with NGFWs from Barracuda Networks, Inc., Cisco Systems Inc., Dell Inc., McAfee Inc. and others.
In giving the lowest rating of any product in this year's review to the Palo Alto NGFW, NSS specifically called out the PA-3020's low overall resistance score -- a mere 65% -- and its low 64% exploit block rate, which dropped from an 80% rating in last year's report.
Palo Alto, meanwhile, hasn't taken the report lightly, calling both the results and the NSS testing methodology into question. In a blog post responding to the report, Palo Alto's Senior Vice President of Product Management Lee Klarich wrote that the Santa Clara, Calif.-based vendor intentionally did not participate in the tests and that decision likely played a role in the negative results.
"Unlike all of the other vendors in the report who configured and tuned their products specifically for this test," Klarich wrote, "there was no input from us on the configuration of our device."
However, NSS Labs Founder and Chief Research Officer Bob Walder responded to Klarich's claims, noting that participation is not a factor in how products are evaluated and that each vendor is treated exactly the same, regardless of whether they make an engineer available to support the testing. While Palo Alto claims that it doesn't understand how NSS could come to what it labeled "a drastically different result compared to the same tests run against the same technology in 2013," Walder noted that Palo Alto's product ran PAN-OS version 4.1.9 in last year's test, while this year's product used version 6.0.3, and that specific changes in the newer platform contributed to the susceptibility to multiple evasion techniques.
Klarich also accused NSS Labs of adopting a "pay-to-play" approach -- something NSS Labs refutes. In fact, Walder noted that it is not Palo Alto's first time accusing NSS of malfeasance; the vendor previously apologized for making similar comments about NSS Labs in the past.
However, Palo Alto isn't the only vendor that has had harsh words for NSS Labs following unfavorable test results. Back in April, FireEye Inc. CTO Dave Merkel condemned the results of NSS Labs' Breach Detection Systems Comparative Analysis and Security Value Map, and criticized NSS Labs for "flaws of the testing methodology." In response to that situation, NSS Labs' CEO Vikram Phatak said, "FireEye's product simply did not perform as well as those from other vendors."
Walder admitted that while the Palo Alto NGFW wasn't the only product that fared badly, it was the only vendor that responded by "attacking NSS in public." In a show of good faith, NSS Labs has offered detailed information about its findings to Palo Alto customers for free in order to "allow the company's customers to at least attempt to improve the protection offered by PAN-OS devices to the best of their ability, until such time as Palo Alto Networks decides to take this issue seriously."
Independent testing has become a critical part of the product-evaluation process for potential and current customers. Still, a skeptic might say that the NSS Labs test is merely one data point about the effectiveness of today's NGFW products, and NSS Labs clearly has benefitted from the media attention generated by controversial test results. On the other hand, customers of both FireEye and Palo Alto might very well be disappointed that the vendors have focused their efforts on engaging in a public war of words instead of demonstrating their commitments to improving their respective products.
In other news
- Researchers from Offensive Security successfully bypassed the protections offered by the latest version of Microsoft's Enhanced Mitigation Experience Toolkit (5.0) after having previously bypassed EMET 4.1 earlier this year (Bromium achieved the same feat and worked with Microsoft to remediate the issues). EMET 5.0 was released in July of this year. In its blog, Offensive Security researchers wrote that they were "curious to see how different it would be to adapt our previous disarming technique to this new version of EMET." The result? The researchers found "the difficulty in disarming EMET 5 mitigations has not increased substantially since version 4.x."
- Security consultant, author and convicted hacker Kevin Mitnick has entered a new line of business: selling zero-day exploits at a hefty price tag of $100,000 each. The black hat hacker was convicted in 1999 of multiple crimes after hacking the networks of some of today's largest companies, and, after spending five years in prison, has since sought to recast himself as a white hat hacker looking to use his skills for the benefit of the security community. In this newest business venture, "Mitnick's Absolute Zero Day Exploit Exchange," clients can purchase a variety of previously undisclosed exploits, though Mitnick claimed, "I'm not interested in helping government agencies spy on people." While some are up in arms about Mitnick's offerings, it is not illegal to sell zero-day exploits.
- The self-acclaimed "one-stop shop for WordPress themes, plugins and training" website iThemes Media LLC reported last week that after detecting suspicious behavior on its site, it found hackers had infiltrated its membership database. While users were urged to change their passwords immediately and sent temporary password resets, it took another two days for iThemes to reveal that it had been storing members' passwords in plaintext, putting the email addresses, names, IP addresses and other critical information of the site's 60,000 users at risk. The company admitted that it did not protect the passwords as it should have, and iThemes Founder and CEO Cory Miller wrote on the company's blog that it has "made the migration process our primary focus, specifically the management and storage of all personal data -- including passwords."
Get in on the discussion about NSS Labs' previous controversy with FireEye.