Logs are the records of what's happening on your networks and who is doing what with your applications. But, too often they sit in silos throughout your organization, sometimes without the technical expertise and/or the relevant business knowledge to separate the wheat from the chaff among the gigabytes and terabytes of information that rapidly accumulate before anyone realizes there may be trouble.
Regulatory pressure, particularly PCI-DSS, is pushing log monitoring, management and archiving from nice-to-have/probably should have status to got to have.
That's pushing corporations to move from ad hoc monitoring or in-house solutions to log management products and, increasingly, to a growing choice of log management services, catering to both mid-market companies and large enterprises.
"First and foremost, it's a security requirement based on principals I try to enforce at GSI," said Wyman Lewis, director of information security at GSI Commerce, which provides e-commerce services to businesses from Toys 'R Us to the National Football League. "Logs are a lifeline of troubleshooting, to see what really happened, not just breaches but any issues. Number one, it's just a matter of doing business, keeping our business going in case something happens. Compliance is a close second.
GSI uses log management service from AlertLogic, one of a handful of pureplay providers. More commonly, providers, including large telcos, are offering services based on product technology. LogLogic, for example, provides the platform for Verizon Business' application log monitoring service, SecureWorks log retention service and Savvis' log management services. HP uses SenSage for its service offering (as well as an OEM deal for its log management appliance).
High-end SIEM vendors like ArcSight and RSA (formerly Network Intelligence), have seized on this market to offer their own log management products, aimed generally at large enterprises.
Both the product and service markets appear to be growing swiftly, but some vendors, always optimistic sound positively giddy when they discuss services.
"Services are growing faster than the product business, and product business grows 100 percent year-to-year," said Dominique Levin, vice president of products, business development and marketing at LogLogic. "There's an incredible demand. Growth rates are phenomenal across the board, primarily in the mid market and enterprise."
Service provider Sage Data Security hopes that's true.
"We focus on the mid-sized market. Fortune 500s have staff for log management; they don't, and it is expensive," said Sage president Sari Greene. "We can apply knowledge over and over again to logs we are looking at."
Both businesses and vendors cite security concerns--with the latter's obligatory reference to whatever major breach is currently in the headlines--for the rising interest in both log management products and services, but there's little doubt that compliance in general and PCI in particular is freeing most of the money.
"Our SOX auditors look into our security controls, and especially for PCI, we need to make sure logging enabled," said GSI's Lewis. "We need to make sure we can show these guys we are logging effectively and retaining logs.
"Our partners are regulated by SOX and PCI as well," he said. We can give them assurance that we can provide information that shows we are compliant. Their compliance requires that third parties must meet certain requirements that we provide."
Faced with the need to retain logs, companies are typically archiving everything for at least a year, often in the absence of any well-defined policy for identifying and keeping what is truly important. As a result they face storage challenges--collecting and retaining logs, moving them efficiently to SAN- or NAS-based mass storage as needed/
"We want to keep everything," said Jim Lairmore, vice president and information security manager of Southern California's PFF Bank & Trust, a Sensage customer, who started with a few Windows servers and now collects logs from some 150 devices of all sorts.
Lairmore said one of the main drivers was that you could put any type of log source into the system.
"We funnel everything into it that we can: anything that can be syslogged, our routers, switches, our firewalls, all of our windows logs, email gateway, VPN, DHCP. Pretty much anything we can get in there," Lairmore said.
"Sometimes auditors want logs from previous year," he added, "but on the source systems it's gone. Now we have four years of data stored on site."
PFF finds on-site storage efficient and cost effective, using a five-node RAID array cluster, but, for some companies, off-site storage offered by managed services is a compelling incentive, as the storage requirement becomes an issue in itself.
"We weren't specifically looking for service solution, we looked at products as well said Lewis. "But that meant we had to manage it ourselves, purchase additional disk space, install agents--introduce another application into a very complicated infrastructure. The service-based solution is cost effective."
Many corporations, which may have perused logs when they could or when an incident sparked an investigation, face requirements to examine them every day. Consider what that means in an environment with network and network security devices , from switches and routers to firewalls and intrusion detection/prevention systems, and a mushroom field of commercial and home-grown applications, assorted database platforms, etc. At the every least, they need some way to distill that all down to the most important information in a usable format. In many cases, they may need someone else looking at the data and alerting/reporting on what is urgent or important.
"Less than 1% of logs are of interest," said Mike Reagan, vice president of marketing at LogRhythm. "You need to store all of the log data and preserve it in a manner that makes it easy to drill down."
That's asking a lot.
"An OTS audit finding four years ago wanted IDS on our network, but it make sense to collect all this data without having any way to correlate it, sort it and filter it," said PFF's Lairmore. "We wrote our own custom stuff, but there was no way to sort it. Pulling out an event ID wasn't easy. It was very reactionary and sometimes took days or a week. Now, it's still reactionary, but we get to it far more quickly and correlate between multiple machines."