Information Security

Defending the digital infrastructure


Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Striving for better information security intelligence

Security teams strive to gain visibility from a deluge of security information and put that data to work.

Getting a handle on “big data” has become a priority for every aspect of business, and information security is no exception. The number and variety of threats and vulnerabilities have exploded in recent years. Add in the challenges of managing exposure in complex IT environments, and the burden on security organizations just to stay abreast of it all can be overwhelming.

Yet security’s own data deluge also conceals tremendous potential, for in this mass of information is the evidence that could help businesses better understand the reality of their security posture and manage risk more effectively. How can security teams turn big data from a threat into an opportunity?

This is the objective of security intelligence, a term that is undergoing a re-definition on multiple levels. As the number and variety of potentially rich data sources has expanded, security professionals are striving to develop disciplines and expertise that yield more actionable intelligence. Security technologies also are turning to more dynamic data sources to drive their functionality, in the beginnings of a trend that may transform the nature of IT defense. As these initiatives gain momentum, innovations from related domains such as business intelligence and the rise of what Turing Award recipient Jim Gray calls the “fourth paradigm” of data science, may become more visible in the world of IT risk.

The changes being wrought by an abundance of information and the demand for greater insight into the reality of information risk are already well underway. How security teams learn to make the most of both the opportunities and the challenges these changes present may transform what the practice of security becomes tomorrow, as a focus on “data-driven security” shapes the way information turns to insight -- and action.


The need to tame runaway data is just one issue facing security teams. A far more significant factor,  particularly for organizations responsible for high-value information assets,  is an increased recognition of the economics of exploit. Those who know their assets are a target also know that dedicated attackers have the means to evade many common defenses. Nick Selby, a Texas police officer and managing director of, says, “If it costs you 50 million to build intellectual property but only 3 million to steal, [what attackers] can say no to numbers like that?” The need for realistic awareness of the threat landscape -- within the enterprise and beyond -- has led many businesses to expand their investment in both broader insight outside the organization, and deeper visibility within.

Large enterprises aren’t the only organizations that have become more sensitive to information risks. According to Rick Howard, general manager of Verisign’s iDefense security intelligence unit, a growing awareness of more sophisticated techniques for getting at sensitive data or threatening business priorities has heightened the need for security intelligence across a broader spectrum of organizations than in the past. “Cyberwarfare, espionage -- large companies and governments have known about these threats for a long time,” says Howard. “But when it comes to things like hacktivism, reputation risk or the more dedicated adversary, there's a much larger swath of businesses worrying about these today than ever before.”

Within the enterprise, businesses need to improve their performance with the information they already collect. In the 2009 Verizon Data Breach Investigations Report, evidence of a breach was available to the victim organization in 82 percent of cases, but was either not noticed or not acted on. That figure actually increased to 86 percent in Verizon’s 2010 report. In its 2011 report, 86 percent of breaches were discovered by a third party. Many companies recognize that their current approaches will not address these failures, and are looking for ways to make better use of the information they collect.

Many businesses also recognize that potentially meaningful data may be locked up in silos of information that, if better used, could reveal the nature of threats more fully. For example, marketing data says much about behavior across entire populations, while within the business, logistical and facilities data can reveal where physical activity corresponds to a digital threat. Executive protection services may correlate suspicious behavior in IT with a specific target, while resources from social media to brand protection services can be used by defenders to identify potential threats before they materialize.

These factors suggest how the drivers of security intelligence may have much in common with those behind the growth of business intelligence (BI) for yielding valuable insights out of large volumes of often disparate data sources. Not surprisingly, more than a few major enterprise technology vendors such as EMC, HP and IBM have increased their investments in both security and BI in recent years. These two lines of business may not have much in common -- yet -- but the opportunity is there. The converging interests of security and areas such as data analytics suggest where they may intersect tomorrow, as the demand for security intelligence drives the need for better performance in information management, and as linkages grow and expand across data sets that, on the surface, may seem unrelated.

A focus on technology, however, can distract security organizations from where investment in intelligence may be needed most.  “Intelligence is a discipline. It may be a noun that describes what you get, but ultimately it is a process. The problem is not with getting information disseminated. It’s with extracting meaningful, actionable truth,” Selby says. This highlights the need for organizations to develop both individual expertise and organizational processes for making the most of security data.


IT security product vendors and intelligence services have long provided current information on vulnerabilities and attacks to their clients. With increased growth and diversity in adversaries and threat tactics, these services have expanded their range. Providers such as Cyveillance, Verisign’s iDefense business and Vigilant’s recently introduced intelligence services enable organizations to extend their eyes and ears into domains outside their normal reach. Subscribers often benefit from information available only to those with specific expertise, credentials or ability to participate in restricted groups or organizations. Providers may offer access to in-country foreign expertise or specialized insight into brand or reputation risks that may be hard to come by otherwise. Intelligence services may also play to their strengths, as with iDefense insight that supports assurance for critical services such as DNS, thanks to its relationship with parent Verisign.

The ability of intelligence services to connect threats against IT with physical risks has become particularly valuable in industries such as utilities, where these interests often intersect. Lester “Chip” Johnson, a 28-year veteran of South Carolina law enforcement and security for the energy sector who now heads his own security and risk management consulting firm, says, “You could literally have your own staff doing just this alone, full time. No one’s budget will allow for that, yet utilities and energy companies can’t do without this sort of information in today’s world. Intelligence services help meet that need.”

As the availability of information increases, so does access to what some regard as “open source” intelligence. Security professionals are always seeking new ways to obtain data. “We’re always trying to get as close to the source as humanly possible,” says one security analyst for a financial services firm who, though his organization subscribes to intelligence services, relies even more on his relationships with leading researchers, community-based efforts, and information that can be collected directly by his organization.

The ubiquitous information gathering capabilities of search and social networks are also having an impact on security intelligence. Businesses increasingly look to sources such as Google and Facebook for background on prospects, partners and job candidates -- a trend not lost on security intelligence services, which also use these resources in gathering broader information on the risks their clients face. And more than a few security pros rely on Twitter for keeping up with the latest thinking among their peers.

One increasingly visible source of intelligence is behavior observable directly from network and application activity. Fraud data collected in this way helps defend tangible assets against exploit in banking, for example. Within the enterprise, technologies such as those of AccessData’s SilentRunner, NetWitness (recently acquired by EMC), Niksun and Solera Networks enable organizations to record complete network traffic content for threat investigation and forensic analysis. At the service provider level, companies such as Narus (acquired by Boeing) can derive intelligence from observation of network content on carrier-class networks, while others, such as Verint, provide analytic capabilities for complex network content such as voice, video and unstructured text.

It’s worth noting that vendors such as Narus originated in network usage analysis to help service providers develop more profitable services. This echoes other ways in which network management technology has become valuable in security, as with anomaly detection technologies used in intrusion detection and security event management systems that use NetFlow data originally intended to support network management for applications. This suggests the role other vendors with visibility into network activity may play in delivering intelligence services in the future. Renesys, for example, provides data on how network traffic actually moves throughout the world. Today, this capability helps ISPs -- and their customers -- keep a closer eye on how network traffic is managed in compliance with service agreements, for example. Tomorrow, this capability may find useful application in providing insight into network traffic movement with direct implications for security.

Many data sources focus on what may be considered “inputs:” potential threat sources, attacks in the wild or exploitable vulnerabilities. In the last few years, outcome data has added a dimension of realism to security intelligence that was largely lacking. Few examples highlight this more than information on data breach cases provided by those such as the Open Security Foundation through its DataLossDB the Privacy Rights Clearinghouse, and Verizon’s annual Data Breach Investigations Report. These resources reveal how compromise actually succeeds across multiple incidents, highlighting commonalities and calling out trends.

Security Intelligence and “Big Data”

So far, discussions of “big data” in the realm of security have largely to do with the sheer volume of data that confronts security teams. Outside the field, “Big Data” (note the capitals) may more frequently invoke large-scale data analysis, data warehousing and mining, and tools such as multidimensional databases and the increasingly recognized toolsets such as  Hadoop and techniques for optimizing large-scale analysis such as MapReduce. Do these two notions of big data ever intersect? Do the techniques of large-scale data analysis have potential for application to security and intelligence?

Though they still have a ways to go before these intersections become common, the answer is a resounding “yes.” Consider, for example, that not only can columnar databases such as Hadoop’s HBase and Google’s BigTable scale out enormously, they may also free data management from the need for schemas or pre-existing structural definitions. Schemas can be generated when data is retrieved, rather than relying on a schema defined when the database is initially set up. This means that, when new types or formats of data are introduced, they can be added to the database virtually as-is. This is a particular benefit to security and intelligence data, which may include such things as malware variants or a wide variety of digital evidence -- some perhaps not previously used in intelligence gathering.

Consider also that techniques of distributed analysis such as MapReduce and highly scalable storage strategies such as the Hadoop Distributed File System (HDFS) not only enable massive scale for data management, they also provide techniques for improving the performance of analysis by assuring that computation is carried out as close to the data as possible. And this, in turn, more closely fits models of cloud computing that make highly responsive, large-scale data analysis for time-sensitive tasks such as intelligence gathering far more accessible to a wider range of organizations than many legacy approaches.


One of security’s biggest challenges is turning data not only into understanding (which is, of course, implicit in the very meaning of “intelligence”), but into action.

Intelligence services increasingly focus on tailoring the delivery of information to meet specific customer requirements. Secunia’s Vulnerability Intelligence Manager enables organizations to customize vulnerability alerts that target those most directly responsible for handling specific systems or vulnerability remediation processes within an organization. Cyveillance, iDefense and Vigilant support the integration of their feeds directly into centralized platforms for managing security data, such as security information and event management (SIEM) consoles, vulnerability management platforms such as Qualys, and systems for managing IT governance, risk management and compliance (GRC) priorities.

Recent years have also seen the rise of data synthesis platforms such as Maltego and Palantir that give investigators tools for connecting data points and visualizing analysis in ways that make understanding more vivid. Forensic analysis platforms have also expanded their ability to develop data synthesis, combining tactical intelligence gathering from the network with input from multiple data sources and creative approaches to data visualization.

Intelligence is also being put to work more directly in the technologies of defense. Legacy defense technologies may be considered “data-consuming,” caching local copies of static data such as threat signatures that are updated intermittently. In contrast, today’s emerging defenses may be considered more “data-driven,” with more dynamic dependence on data feeds or behavior observed in real (or near-real) time.

Distributed threats such as botnets, for example, can be identified through techniques such as recognizing when compromised systems attempt to communicate with known malicious command-and-control hosts. Team Cymru’s BATTLE service for law enforcement and  companies such as Damballa, FireEye, Umbra Data and Endgame Systems’ recently introduced ipTrust products enable customers to leverage their knowledge of botnet behavior gained across multiple enterprise and service provider networks. Using these tools, businesses can identify -- and in some cases, block -- specific incidents of compromise based on current insight into fast-changing botnet topologies.

Fraud prevention technologies also leverage insight into observed behavior. Risk-based authentication techniques use this data directly to increase the level of authentication required or deny access altogether when attempts to access sensitive resources appear to be anomalous, such as access attempts from two distant physical locations within a few minutes of each other. Silver Tail Systems monitors website traffic over time, recognizes attempts to abuse or manipulate application behavior, and enables organizations to mitigate risk by modifying business flows in Web applications. Mykonos can both identify and track malicious parties that seek to exploit Web applications while defending the underlying application itself.

Antivirus and antimalware technologies increasingly look to data sources to improve performance in recognizing new attacks. When suspicious content is not specifically recognized by security software on the endpoint, its characteristics can be referred to centralized resources for more immediate analysis and faster deployment of defense. These approaches leverage insight into the prevalence of software gathered from among their global customers to help differentiate threats and bring the identification of new attacks to all customers more quickly. They help close the “zero-day window of opportunity” and reduce the footprint of security software at the endpoint. Such techniques are already reshaping the nature of antivirus and antimalware, as exemplified by Symantec’s Insight, McAfee’s Global Threat Intelligence, and Trend Micro’s Smart Protection Network.

The Risks of Intelligence

Today’s information explosion can be a potent enabler of security initiatives, but it has its risks. First off, the tools that enable security teams to better understand malicious activity may also be just as available to the adversary. Search, for example, is already recognized as a technology that can reveal more personal and corporate information, – including security weaknesses, than many would care to acknowledge.

The explosion of social networks has amplified these concerns. “One of the biggest changes we’ve seen in the last year or more has been the use of social networks by criminals, not only through malicious URLs, but also to discover more personal information about targets to elicit a specific response,” says James Brooks,  Cyveillance director of product management. He cites the openness of social networks that leads many to assume a false sense of security. Others point to the difficulty of keeping up with a broad, intricate and often changing scope of privacy settings on sites such as Facebook.

Civil libertarians and those who advocate for greater access to the information collected by governments and other organizations have spoken out, not just about intelligence gathering in the name of security, but also about risks such as the relentless collection of personal information by search and social networks. In a recent interview with Russia Today, Wikileaks founder Julian Assange called Facebook “the most appalling spying machine that has ever been invented.”


Greater insight poses risks of its own. Intelligence can be an advantage to the adversary as well as to the defender. Concerns about both the scale and the nature of information collected in the name of intelligence have motivated some to take direct action to expose what they see as abuses. (see “The Risks of Intelligence”)

Regardless whether intelligence is seen as an asset or a threat, there seems little doubt that the growing abundance of data available to security teams will continue to drive the demand for greater insight into risk. In order to make the best -- as well as the most conscientious -- use of this data, people will need to develop the individual skills and the organizational practices that define the ultimate meaning of intelligence.  Police-Led Intelligence’s Selby says, “There is an ‘aliveness’ to intelligence that people often overlook. Intelligence must be iterative, with constant feedback, checks on bias, constantly evaluating and re-evaluating sources and calling conclusions into question.”

Those on the cutting edge of security information analysis would likely agree with the human-centric nature of intelligence. In fact,  one such professional says, “Technology can clear away a lot of the noise, but you must first make it understand what is noise, and what isn’t. Automation can’t make any difference until we tell it what to care about, and we may not know what to care about until we’ve investigated.”

This suggests the rise of a new approach to security practice, one where defense becomes a function of visibility, and where automation is more dynamically and responsively defined by investigative expertise. The value of the practice of investigation within security organizations has already been called out by examples such as EMC’s acquisition of NetWitness. Though a technology play, NetWitness primarily serves investigators who know how to use its tools. Its output, however, can be used to refine the automation of security information analysis, not just in correlating security events, but in mining security data to discover concerns that might otherwise go unnoticed.

Ultimately, this trend may lead us in directions foreseen by those who see the rise of data science as a new profession. What security becomes in the future may well depend on the expertise being built today by those directly engaged not only in the work of intelligence, but in the still-emerging discipline of investigation based primarily on data analysis. Together, this fusion of expertise may well define the data-driven security strategies and tactics that will make a difference in what security intelligence becomes tomorrow.

Scott Crawford is managing research director of security and risk at Enterprise Management Associates, an IT industry analyst firm based in Boulder, Colorado. He is the former CISO of the Comprehensive Nuclear-Test-Ban Treaty Organization’s International Data Centre in Vienna, Austria, and has over 20 years’ experience as an IT security and operations professional in the private and public sectors, with organizations including the University Corporation for Atmospheric Research and Emerson. Send comments on this article to

Article 4 of 7

Dig Deeper on Security industry market trends, predictions and forecasts

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

Get More Information Security

Access to all of our back issues View All