Enterprise information security teams have long been intrigued by big data analytics tools like Hadoop that other IT sectors have successfully used to analyze marketing campaigns or optimize business operations, but actual adoption rates among infosec teams have lagged.
However, new research shows a growing number of enterprises are either already using big data analytics for security purposes, or are exploring ways to do so.
For its recently compiled 2014-2015 Enterprises Technology Benchmark on Big Data and Security, , Mokena, Ill.-based IT research and advisory firm Nemertes Research Group Inc. interviewed more than 200 IT practitioners including CISOs, administrators and other security professionals to determine whether big data is being used for security programs. Participants spanned all industries and skewed slightly toward larger organizations, meaning those with more than 10,000 employees and $1 billion in annual revenue.
The results showed that enterprise interest in big data security analytics is definitely on the rise. Nearly a quarter of the benchmark participants said that their respective organizations already use big data for security, while another 13.6% said plans are in place to implement big data tools by the end of this year.
Furthermore, almost one-fifth of participants said that they are in the evaluation phase. Johna Till Johnson, CEO and founding partner for Nemertes, said she expects most of that group will eventually plan to deploy by the end of 2015, though it is still too early to determine how many will do so.
Big data security analytics: Why now?
As for why interest in big data security is increasing, Johnson said that a number of factors converged together in 2014 that created a "perfect storm" for survey respondents. For one, attackers are increasingly capable of deploying sophisticated attacks that can bypass traditional enterprise defenses such as firewalls for extended periods, even at organizations with well-funded security teams.
Simultaneously, security breaches are more likely than ever to affect CEOs and other C-suite executives, according to Johnson, as was the case when former Target CEO Gregg Steinhafel and CIO Beth Jacob were ousted after the company experienced one of the largest retail data breaches ever.
And, of course, the big data tools themselves have become more adept at handling the needs of security teams, she noted -- Splunk, for example, offers an app to its enterprise customers that can correlate threats across various data types and sets.
"Attacks are coming from all different directions, and you can't really sit there and reverse engineer it and say, 'This indicates an attack,'" said Johnson. "But big data can do that for you."
Vijay Dheap, global product manager, big data security intelligence at IBM, said that security has long dealt with big data issues such as analyzing troves of log data for relevant information. Broader use cases have often lagged behind other business sectors though, Dheap said, because security is the only IT segment that must deal with adversaries and, as a result, data that changes constantly.
Beyond Johnson's points, Dheap said that interest in big data security analytics is now surging thanks to two relatively new factors. One, he said enterprises are now committed to storing security-relevant data for not just the usual 30 to 60 day period, but up to a year or longer.
According to Dheap, the other element in play is that enterprises want to move beyond analyzing current data and instead look to historical and predictive analytics to spot attacks before they ever happen.
"If I have a thousand vulnerabilities, which do I work on today and which do I leave for tomorrow?" asked Dheap. "These types of analytics are what enterprise customers are looking for."
Know-how, not money, holding back big data security analytics
While the security use cases for big data are becoming more defined, nearly half of participants in the Nemertes benchmark research still have no current plans to even evaluate big data tools. Funding doesn't seem to be the problem, according to Johnson, noting that all of the participants Nemertes interviewed said their respective organization plans to increase spending for enterprise risk management in 2015.
Instead, Johnson said by far the most pervasive factor that holds back enterprise big data security analytics is a lack of qualified professionals with a unique combination of talents, namely people with expertise in both big data and security.
"We have dedicated tools for leakage detection (IBM, RSA). We're looking at what we can do, currently doing log analysis," said one participant in Nemertes' study, according to Johnson, "but it's hard to find someone who's a data scientist focusing on big data and who can also understand all the various logs."
Along similar lines, Dheap said that many enterprises start with big data security analytics by cobbling together open source technologies, but often fail to define the roles of security team members in the project or nab the big data scientists need to make sense of the incoming data. As a result, he added, organizations are often left confused about what their chosen analytics platforms can offer them.
Though Nemertes declined to provide supporting statistics, Johnson said that some big data teams and security professionals may not have considered the security ramifications of storing large repositories of potentially sensitive data. In regulation-heavy industries such as healthcare, which is governed by HIPAA, experts have indicated in the past that privacy and protection requirements for health data may reduce the use cases for predictive analytics.
"A few folks indicated that they hand off [responsibility for protecting the data] to the privacy teams after compiling the data, but nobody's mentioned that they are taking extra-special precautions," said Johnson. "I suspect they are, but I haven't asked, and nobody's raised the issue."
Read about one of the earliest big data security analytics implementations at Zions Bancorp.
Still wondering whether big data analytics are more hype than reality when it comes to security? Read this conversation between security luminaries Marcus Ranum and Gartner research director Anton Chuvakin where they discuss big data and security.