Manage Learn to apply best practices and optimize your operations.

Security event log management, analysis needs effective ways to search log files

Search is a key discipline for security log management. John Burke explains how to better search log files to improve security event log management.

This tip is part of's Integration of Networking and Security School lesson, Security event log management. For more learning resources, visit either the lesson page or the Integration of Networking and Security School main page.

To examine where enterprise information security log analysis practices are today and where they must go, it's helpful to take a brief step back in time. Interestingly enough, some of the best methods to help ensure thorough security log analysis in the future are the same concepts that have worked in the past.

The notional evolutionary path for security event log management runs like this: Old Stone Age Security Admin manually reviewed logs, looking for bad behavior, misconfiguration and bad luck; crafty tool-user New Stone Age Security Admin added scripting to automate the review; later, non-homemade tools from others automate the scanning previous generations did manually.  Over time, crude, not so sharp flint (home-brewed Perl scripts) becomes razor-edged obsidian (Snort), becomes, for Industrial Age Security Admin, a mass-produced steel axe (insert your favorite full-blown SIEM system here). 

In today's security data-processing era, log files are the commoditized inputs for mass production of alerts driven by rules-based log analysis and scanning.  This is indispensible as a foundation, but insufficient in itself. After all, attacks pop up that surprise us all, succeeding by nailing exploitable flaws in platforms or taking advantage of inconsistent and incomplete security configurations.  You set up rules and alerts for specific patterns (known bad things), but the unknown bad things get in anyway because there was no pattern to detect them.

What is Post-Modern Information Age Security Admin to do?  

The answer, so to speak, is to go back to the Paleolithic! Using the full arsenal of aggregation, indexing and automated pattern-matching analysis as a base, make use of one of humanity’s oldest and most deeply embedded skills: pattern matching.  The best security combines well-rehearsed and consistent procedures, solid instrumentation and automated processing with ad-hoc, free-form security review.    Empower admins to swoop through log data using a non-deterministic and flexible pattern matching machine—the brain—to match the deeply non-deterministic and flexible world of evolving security threats.  Humans are still better than any rules-based system at finding things that are just “weird” even without knowing what constitutes “weird” in advance; skilled security folks just know it when they see it.

So, IT needs to be supplementing the event processing factory of automated log analysis with robust search tools to help security engineers conduct the post-industrial version of a hunt, searching for meaningful signs amid the scrub and shadows, but with power tools, the equivalent of search lights and rifle scopes and motion detectors.

A search tool with full access to all the same streams of data that automated analysis tools use – system logs, application logs, etc. – lets security staff dive in and see anything that just doesn’t look right, then explore and see if there is a pattern there to discover.  It is essential, of course, that a search tool supporting security has the ability to store a search for use again later, and to turn an ad-hoc search into an automated one, creating a new pattern for the baseline tool to run constantly.

When it comes time to search log files, the best tool in the world (or the best automated analysis, for that matter) won’t find events to worry about in data it doesn’t see.  So it is important to feed in as many streams as possible. In addition to OS, middleware, application logs and network security appliance logs, also feed it system configuration files, CMDB data and system management scripts.  Moreover, make sure logs come not just from the physical servers and appliances, but also from the virtual ones too, and from systems running in a cloud infrastructure, not just on-premise.  This can take some traffic engineering to ensure log traffic doesn’t get in the way of production traffic, but failing to include off-premise systems in the mix leaves you with huge blind spots.

Blind spots can be human and organizational, too, so it is ideal that security tools be able to search through data across all organizational and technological silos.  Especially in an age of adaptive persistent threats (APT), and with penetration efforts increasingly multipronged and multimode, tools need to be able to see from the bottom of the network stack to the top, from Ethernet to application.  It helps if network operation and security operation centers (NOC and SOC) merge; a unifying tool can be the first step in implementing that kind of organizational convergence.

Put another way, when hunting for a needle in a haystack, it’s simple to get a tool to solve the problem: a magnet, or a metal detector, or a match.  When hunting for a needle you’ve never seen in a stack of pins, you need to get tools that help the human eye and brain see through the pins, that automatically get rid of the needles you already recognize, and help you spot the new needle they all obscure.  Search is that kind of tool.  To get the most out of it, embed it in an environment with rich automation, endow it with the ability to feed into that automation, and let it help surmount and even dissolve silos that interfere with proper log analysis, be they technological or organizational.

About the author: John Burke is principal research analyst with Nemertes Research. With nearly two decades of technology experience, he has worked at all levels of IT, including end-user support specialist, programmer, system administrator, database specialist, network administrator, network architect and systems architect. He has worked at The Johns Hopkins University, The College of St. Catherine, and the University of St. Thomas.

This was last published in March 2012

Dig Deeper on Data security technology and strategy