Problem solve Get help with specific problems with your technologies, process and projects.

Review system event logs with Splunk

Splunk is a free tool that provides log review and management. From parsing files to triggering alerts and scripts, Splunk can greatly reduce the amount of time security teams spend on logs.

Logging and log review are two of the most difficult challenges faced by security pros. Even if a person is dedicated to nothing but log review, he or she would be quickly overwhelmed by the volume of information and the tedium of the task. That's why dedicated log-review systems are a must, and there's a tool for the job that's simple but powerful -- and free.

Splunk is a great tool for grabbing all sorts of network data, making it simple to search, providing triggers and alerts, keeping the data secure with granular access controls and offering controls for data audit and data integrity.

Splunk allows a variety of inputs, including logs, configuration files, traps and alerts, device and system messages, scripts, and performance data from applications, servers and networked devices. The software monitors file systems for configuration changes, watches files and logs, and can connect to network ports to receive syslog, SNMP and other network-based data. The Web interface uses drop-down boxes, making it easy to select a file to monitor, such as an actively growing log file, showing the most recent updates first.

Point Splunk to a file and it will intelligently parse the file, working out the event-types and normalizing a multitude of timestamp formats across different log types. Data is parsed and indexed on-the-fly, while raw events are kept for review. The data is secured with an MD5 hash using a PKI signature to detect tampering and point out gaps in the log where specific data may have been deleted. The tool keeps an audit record of who administers the system and who accesses data.

Splunk supports free-form search, but has a few tricks up its sleeve, such as a pop-up window with common events and values from logs to allow the administrator to zero in on specific behaviors, such as server errors.

Searches can be run on a schedule and set to trigger notifications or actions based on search results. Alerts can be used to monitor user or system activity and can trigger based on event types, event source or even the number of events; alerts can also trigger scripts to perform an action, such as restarting an application or service when it detects a condition. Notifications can be sent via email, RSS or Simple Network Management Protocol (SNMP) to other management consoles.

For security, Splunk uses SSL over TCP to secure the data-path. User sessions with the browser are performed using HTTPS over SSL. Roll-based access supports multiple types of users and comes with pre-built roles that can be modified or supplemented. Splunk integrates with Active Directory, LDAP and other directory services.

The biggest difference between the free version of Splunk and the commercial version is a cap that limits the maximum indexing volume to 500 MB per day. While meant to entice users to try Splunk, this volume cap may suffice for many small organizations.

Splunk works across a variety of platforms and deploys fairly quickly. It has an intelligently thought-out interface, which makes it relatively easy to use. If your company is looking for a product that offers log collection, review, searching, parsing and alerting -- for free -- Splunk may provide a smart and secure way to get the job done.

More Information

  • Learn about the latest efforts to develop a common logging and audit standard.
  • Interested in how to mine enterprise SIM logs for relevant security event data? Read more.

About the author:
Scott Sidel is an ISSO with Lockheed Martin. For more recommendations from the author, check out Scott Sidel's Downloads.

This was last published in November 2008

Dig Deeper on SIEM, log management and big data security analytics