News Stay informed about the latest enterprise technology news and product updates.

Gartner: Big data security will be a struggle, but necessary

The research firm says big data security analysis will be critical to fighting off advanced attacks, but few easy technology options exist today.

NATIONAL HARBOR, Md. – The concept of using "big data" to improve enterprise information security is more than hype, according to a top Gartner Inc. analyst, and will be a reality for many large companies in a few years.

By 2016, 40% of enterprises – led by the banking, insurance, pharmaceutical and defense industries – will actively analyze patterns using data sets of at least 10 terabytes in order to flag potentially dangerous activity.

Gartner Inc.

However, getting to that point may be painful because successful big data security analysis implementations will redraw logical boundaries within many IT departments, and few security vendors today offer capable products to support the transition.

Those were a few of the key messages offered this week by Neil MacDonald, vice president and Gartner fellow, during a presentation at the 2012 Gartner Security & Risk Management Summit. Defining big data as an information-processing problem requiring alternative approaches because of volume, velocity, variety and complexity of the data in question, MacDonald said few IT organizations possess scalable systems to analyze multiple terabytes worth of data at a reasonable speed.

He went on to cite network packet capture, sensors, various types of transactions, compliance monitoring and threat intelligence as a few of the numerous data types that today can't be easily correlated, analyzed and used to make business and information security decisions.

"It's not about the data, it's what you do with it: the insights that come from the analytics of the data you actually care about," MacDonald said. "I believe it's real; it's beyond hype, and this is an example of the information security data we're going to be asked to handle and analyze."

Big data analysis is an increasingly urgent problem for enterprise information security organizations, MacDonald said, because of the rapid rise of advanced persistent attacks, more commonly referred to as APT. Traditional security defenses struggle to detect APT-style attacks because they rarely resemble a previously established malware pattern.

"How do you know if something is bad? You subscribe to vendors' services, they tell you what bad looks like, and you go looking for it," MacDonald said. "Now, how do you find badness in a world where nobody knows what it looks like?"

Instead, enterprises must approach the problem by establishing what normal non-malicious activity looks like, and then seek out deviations. But MacDonald said to do this successfully, an enterprise requires much more data to develop a confident baseline. That's where big data comes in. 

MacDonald predicted that by 2016, 40% of enterprises – led by the banking, insurance, pharmaceutical and defense industries – will actively analyze patterns using data sets of at least 10 terabytes in order to flag potentially dangerous activity.

However, the vendor landscape won't make that evolution easy. Today enterprises typically rely on SIEM systems to correlate and analyze security-related data, but MacDonald said today's SIEM products don't hold up against that type of workload. He said most SIEM products either offer near-real-time data but handle only normalized data. Those that are capable of processing a high volume of raw transaction data can't provide intelligence in anything close to real time.

What that means, MacDonald said, is some enterprises will be compelled to strike out on their own to architect effective big data analysis systems. He predicted more organizations will initiate projects like the Zions Bancorporation big data implementation, in which the company built a Hadoop-based security big data warehouse using technology from startup Zettaset Inc.

Before committing to a custom deployment though, MacDonald recommended reaching out to SIEM vendors and tracking market developments. Some larger vendors, like IBM, Hewlett-Packard Co., and EMC Corp., with its RSA and VMware subsidiaries, are building and integrating similar technologies based largely on their SIEM products. 

Ultimately, MacDonald said, the security big data evolution will be a part of a larger trend called BI for IT, which he described as the combination of information security intelligence and IT operations data to provide a new level of business context.

My experience is most of the vendor solutions don't lend themselves to supplementing the data they give you … so you almost have to do it in-house.

Robert House,
senior director of technical assistance, New Jersey-based firm

MacDonald said there's value in bringing security and operations data sets together because as IT systems become increasingly virtualized, the process of using a baseline of normal behavior to identify deviations will become common in both security and operations. Plus, operations teams will be privy to data that will be crucial to security, such as which the systems that host the organization's most valuable data.

"If you look at the picture I painted … it's going to create a data deluge," MacDonald said. "To get to the top of the pyramid, you have to distill vast amounts of data using meaningful patterns and insights to make it actionable, and know what to do and in what priority. It sounds difficult, but that's the secret sauce. That's what Gartner calls security intelligence: I need to know what to focus on, delivered in the form of an IT risk 'heat map' showing me where to focus my efforts."

Attendees had mixed reactions as to whether this evolution can take place in the next few years. Luis Scull, a research analyst with Open Field Capital in New York, said he doesn't see big data taking hold in the next few years because most organizations don't have the resources or the political capital to make it happen.

However, Robert House, senior director of technical assistance and incident response for a New Jersey-based company, said the topic of a big data security analysis system has already come up in his organization because of the increasing urgency to find innovative ways to detect threats. But SIEM vendors aren't stepping up to help customers take the next step.

"My experience is most of the vendor solutions don't lend themselves to supplementing the data they give you," House said, "so you almost have to do it in-house."

Compliance could drive the big data evolution, House said. There may be more urgency to move toward security big data systems in the coming years as mandates like the Payment Card Industry Data Security Standard (PCI DSS) and the Sarbanes-Oxley Act (SOX) evolve to include more advanced defensive methods.

"With Stuxnet and Flame … as these attacks become more intelligent, you're going to have to prove you're demonstratively in control of your security," House said. "You're going to have to constantly evolve your security methods."

Dig Deeper on Data security technology and strategy

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.