News Stay informed about the latest enterprise technology news and product updates.

For threat intelligence programs, ROI evaluation proves tricky

Threat intelligence programs are taking root in many enterprises, but experts say variables like disparate service offerings, pricing models and response capabilities make ROI evaluation a vexing proposition.

There's a saying that a penny saved is a penny earned. But for CISOs, the challenge they sometimes face is convincing boards of directors that a penny spent on an emerging security technology, like threat intelligence, can produce savings that justify the cost.

When that money is spent on threat intelligence, experts say, articulating ROI [return on investment] can be tricky. After all, if the information prevents a security incident, how does one put a price tag on an incident that never happened?

"I think metrics and being able to measure ROI on threat intel solutions is still in its infancy," said Damon Rouse, director of information technology at San Diego-based professional services firm Epsilon Systems Solutions Inc.

Rouse's organization sought a way to gain better understanding of its adversaries and their motives, ideally in a more proactive way. That led him to explore threat intelligence technologies.

Epsilon eventually chose to implement ThreatConnect, which provides a threat intelligence platform that aggregates intelligence from multiple sources, including open source indicator and reputation feeds, as well as vendor-supplied threat intelligence information.

"For us, we looked at things from a more organic perspective," he said. "We asked what could potentially happen to the company if we suffered a compromise and based justifying the spend on this. The negative effects to the business from a potential compromise were way more than the expenditure needed for a solid threat intel solution."

ThreatConnect is one of dozens of vendors pitching threat intelligence in a nascent market segment that includes companies ranging from Dell SecureWorks to iSight Partners. Definitions of the market vary and vendors' product offerings often differ considerably from each other, making it critical that organizations understand what they are able to get value from.

Inside the threat intelligence marketplace

According to Anton Chuvakin, research vice president with Stamford, Conn.-based Gartner Inc., deriving utility from a threat intelligence offering hinges not only on the quality of the information, but also on being able to actually use the information received. That way, an organization can rapidly change its security infrastructure, deploy new detection and response capabilities, and ultimately prevent cyberattacks more successfully.

"[Threat intelligence] is not magic: You may receive good signals," Chuvakin said, "but you still have to act on them."

Threat intelligence subscriptions -- most offerings are provided on a continuous basis for an annual fee -- can definitely be useful, Chuvakin added, because there are known examples where companies learned of new attack methods from these data sources and had time to prepare.

Comparing threat intelligence data sources involves looking at the provider's sample data and checking whether it would have helped the organization detect any new threats or respond to incidents better or faster. Geographic and industry-specific coverage can also be factored in before purchasing a threat intelligence subscription.

To be sure, threat intelligence is by no means a set-it-and-forget-it proposition. Wendy Nather, research director with New York-based 451 Research LLC, said that if a company can only afford to have one security analyst dip into the data once a week, there's really no point; the feeds must be monitored closely in order to respond rapidly when needed.

On the other hand, Nather said, if an organization is ready and willing to automate the ingestion of threat intelligence data and use it to make quick decisions, such as creating new rules for firewalls and intrusion prevention systems, it makes sense to consider buying top-of-the-line feeds and to think about collecting more than one set of feeds for a more complete picture.

Adding to the complexity, Nather said, is that the pricing models of threat intelligence services are not one-size-fits-all.

"If it's a data feed, it's generally a subscription," Nather explained. "In most cases, if the provider is selling access to mine that data through a central portal, it's also a subscription. But if they're selling software specifically for analysis and visualization of that threat intel, it can be permanent licenses. Finally, if the threat intel is boutique research, there may be charges for time and effort, or a fixed-price report deliverable."

Quantifying the value of threat intelligence

Since the market segment is still new, few hard metrics exist to prove whether threat intelligence improves incident response and detection, but some advocate demonstrating the value of threat intelligence in a more abstract way when seeking the budget for it.

Threat intelligence doesn't have to lead to a front-page-worthy data breach to be of value. If it helps drive measurable change in how organizations reduce risk and respond to threats, it's a win.
Ryan Stillionshead of detection and response, Vigilant Technology Solutions

Ryan Stillions, head of detection and response at Mason, Ohio-based threat intelligence vendor Vigilant Technology Solutions LLC, equated quantifying the value of threat intelligence to spending 30 minutes with the best legal counsel money can buy.

"We can't really quantify the cost of each comment or statement made, but we leave the conversation knowing we're in a much better position to make informed decisions in the name of managing risk," said Stillions, who recently gave a presentation on leveraging intelligence at the SANS Institute's Cyber Threat Intelligence Summit. "Boards should view their [spending] on strategic and operational threat intelligence in a similar way."

On the other hand, Stillions admitted that quantifying the value of tactical threat intelligence is becoming slightly easier because it rarely takes long for security analysts to determine which sources offer accurate, timely and relevant data, and which do not.

"I like to measure the number of net-new detections over time made purely as a result of having the intelligence … Intel sources who get more scores in this area earn my confidence, those who don't or struggle to establish a baseline over time may require further assessment as to why," Stillions said. "Keep in mind that threat intelligence doesn't have to lead to a front-page-worthy data breach to be of value. If it helps drive measurable change in how organizations reduce risk and respond to threats, it's a win."

Stillings also said that because the market for threat intelligence products and services is still evolving, enterprises should conduct short-term evaluations of multiple vendors before jumping into any long-term commitments.

Sophisticated executives understand the importance of making security investments in order to be proactive as well as how to communicate to executives how business decisions can impact security posture, said Matt Hartley, vice president of product management at Dallas-based threat intelligence vendor iSight Partners Inc.

"If you go to market in certain countries, then you are going to have an espionage campaign targeting you," Hartley said. "A lot of businesses don't understand that. So the sophisticated executive using intelligence is able to communicate [the threat] effectively."

"I think at the most basic level, as long as the threat intel is telling you things you couldn't have figured out by yourself, and doing so on a regular basis, and you're able to make decisions and take actions based on it -- then you're coming out ahead," Nather said. "I've heard at least one group refer to threat intel as 'just another alert to have to chase down,' and if that's all it is to you, then you can probably do without it."

Next Steps

Learn how SIEM products are increasingly integrating threat intelligence feeds.

Dig Deeper on Threat intelligence sharing and services