What do you do with your firewall and system logs? Typically, you collect them and back them up -- and that's about it.
But there are gold nuggets of useful info hidden in those logs that, ironically, many organizations ignore because they incorrectly believe the extraction tools are complicated and expensive. One of the great things about log analysis is that it's relatively easy to implement for a few dozen important servers. Sure, you can spend a lot of money on expensive systems and databases (and many organizations do), but all you need is a desktop PC, a few rudimentary scripts and some patience to get your feet wet.
Think of it this way: How much money does your company spend on firewalls and IDSes? It's probably a lot, right? And yet they collect and analyze the information collected in -- you guessed it -- log files.
Discarding or ignoring logs should be a crime. Think about all the useful intelligence you can draw from this well:
- Do you know how many of your corporate desktops are infected with spyware? Your firewall logs can tell you just by mapping destination Web sites against a blacklist of spyware sites. SquidGuard (www.squidguard.org) maintains an open-source list you can use to create a spyware blacklist.
- Do you ever wonder how many of your desktops are properly updating their antivirus signatures? Subtract the number of addresses that go to the antivirus update site from those going to your firewall, and you'll have good starting point.
- Do you worry that your network is infected with a BotNet? Watch for a spike in IRC traffic.
- Do you dread the next e-mail virus? Look for spikes in your outgoing SMTP connections.
A friend of mine gets more mileage out of his MRTG traffic-monitoring tool (www.mrtg.org) than most organizations get out of their massive IDS farms. He runs most of his apps on an old SPARC Ultra-5 that continuously churns out graphs with various log data. It's front-ended with Perl scripts that pull data from various sources using SNMP queries and secure copy. Total cost: zero. Total setup time: about a week. Hackers caught: three. ROI: gigantic.
The most powerful and simple technique I've seen for log analysis is what I call "artificial ignorance." In this approach, you build a list of syslog messages that you don't care about, and everything that's left over is flagged for review. Setting up a simple artificial ignorance feedback loop only takes a few hours and requires a small amount of daily maintenance. Freeware tools, such as logwatch (www.logwatch.org), are useful in setup (for details, see http://www.ranum.com/security/computer_security/papers).
Always remember my law of log analysis: The number of times an uninteresting thing happens is an interesting thing. If you're going to throw particular pieces of data away, count the number of times they appear and monitor the discard rate. You'd be amazed at the number of interesting things you'll find.
Log analysis is the most under-appreciated, unsexy aspect of infosecurity. Regardless, I believe it's one of the most important. Get over the tedium and start sifting through those logs. You're missing a mother lode of gold.
About the author
Marcus J. Ranum is a senior scientist at TruSecure Corp. and the author of The Myth of Homeland Security (Wiley, 2003).
