Businesses and other organizations operate in a hostile information security environment. It is an environment where compute and storage resources are targets of attackers who would use the compromised systems for malicious operations. Where personal and confidential information is stolen and sold on underground markets, while state-sponsored attacks lead to massive data breaches. In such a situation, it's understandable to think a business should deploy any and all available countermeasures, including big data security analytics, to protect valuable company resources. This technology combines the capabilities of advanced analytics with security event and incident management systems (SIEM), and is appropriate for many -- but not all -- use cases.
Before investing in a big data analytics platform, consider the nature of your environment and the organization's ability to leverage a big data security system. There are several factors to consider here, ranging from the scale of the IT infrastructure to protect, to the marginal cost and benefit of deploying additional security controls.
Scale of infrastructure
Organizations with substantial IT infrastructures are prime candidates for big data security analytics. Applications, operating systems and network devices can all capture traces of malicious activity. By themselves, no single piece of datum or type of data may provide sufficient evidence to identify an active threat; the combination of multiple data sources can provide a more comprehensive view of the state of an attack.
Existing infrastructure and security controls generate the raw data, but big data analytics and analysis applications are needed to collect, ingest and analyze all that information. Big data security analytics may not be needed in environments with small numbers of devices and less complex network structures, however. In such cases, traditional SEIM applications may be sufficient.
Near real-time monitoring
Another factor that drives the need for big data security analytics is the necessity for gathering incident information in near real time. Real-time monitoring is especially important in environments that are subject to advanced attacks and that house valuable data, such as financial services, healthcare and government agencies and contractors.
A recent Verizon study found that in 60% of incidents, attackers are able to compromise systems within minutes, but the proportion of breaches detected within days is significantly less. One way to reduce the time of detection is to collect varied data from across the infrastructure in real time and screen that data immediately for events indicative of an attack. This is a key use case for big data analytics.
Detailed historical data
In spite of best efforts, attacks may occur without detection for some time. In such cases, it is important to have access to historical log and other event data. Forensic analysis can help identify how the attack occurred as long as sufficient data is available.
In some cases, forensic analysis is not needed to identify vulnerabilities or to correct security weaknesses. For example, if a small business is attacked, the most cost-effective remediation may be to engage security consultants to review current configurations and practices and recommend changes. In such cases, big data security analytics is not needed. Other security measures and responses can be effective and less expensive.
On-premises vs. cloud infrastructure
As the name implies, big data security analytics entails the collection and analysis of large volumes of various types of data. Any restrictions on capturing security event information, such as the ability to capture all traffic on a network, could adversely affect the quality of information derived from big data security analytics systems. This is especially the case in cloud environments.
Cloud providers restrict access to network traffic to mitigate the risk of network attacks. Cloud customers, for example, cannot tap network segments to collect comprehensive data on network packets. Prospective big data security analytics users should consider how restrictions imposed by cloud providers curtail the scope of analysis.
There are cases in which big data security analytics is useful with cloud infrastructure, however, particularly with regards to log data generated in the cloud. Amazon Web Services, for example, provides a performance monitoring service, called CloudWatch, and an audit log for cloud API calls, called CloudTrail. Data about operations in a cloud may not be as fine-grained as other data sources, but it can supplement other data sources.
Ability to utilize data
Big data security analytics ingests and correlates large volumes of data. Even when data is summarized and aggregated, it can be challenging to interpret. The quality of the information and insights drawn from big data security analytics is partly a function of analysts' ability to interpret the data. Organizations need security analysts who can follow kill chains, as well as understand network flows and operating system events as they relate to security incidents.
For example, an analyst might receive an alert about suspicious activity on a database server. This is likely not the first step of an attack. Can the analyst start with one alert and navigate through historical data to find correlated events to determine if it is indeed an attack? If not, then the organization is not realizing the benefits of the big data security analytics platform.
Other security controls in place
Organizations also should consider their overall maturity with regards to security practices before taking the plunge into big data security analytics. That is, other less expensive and less complex controls should be in place first.
Well-defined identity and access management policies should be defined, enforced and monitored, for example. Operating systems and applications should be routinely patched. In case of a virtualized environment, the machine images should be rebuilt regularly to ensure the latest patches are incorporated. Alerts to monitor suspicious events or significant changes to the environment (e.g., the addition of an administrator account on a server) should be implemented. Web application firewalls should also be deployed to mitigate the risk of injection attacks and other application-based threats.
The benefit of big data security analytics can be substantial, especially when deployed to an infrastructure that already implements a comprehensive defense strategy.
The business case for big data security analytics
Big data security analytics is a recent addition to information security controls. These systems are designed to consolidate data from multiple sources and reduce the need for manually integrating point solutions. They also address limitations of other security controls, such as difficulties querying across multiple sources. By capturing continuous streams of data from multiple sources, big data analytics systems increase the chance of collecting forensically important details.
Costly, big data security analytics is a significant investment of both capital and staff resources. Consider how a new security platform will integrate with existing security and logging tools. Although there may be some overlap in functionality between a new big data security analytics system and existing controls, that is not necessarily a bad thing. Redundant functionality can help mitigate the risk that one system will miss a potential threat. Organizations with established, mature security practices are subject to government or industry regulations, and maintain complex infrastructures that are likely to benefit the most from the deployment of big data security analytics tools.
In part one of this series, find out about the basics of big data security analytics in the enterprise.
Learn more about the new era of big data security analytics.
Can big data security analytics revolutionize information security?