Published: 12 Jan 2007
Your network devices are trying to tell you that you're under attack. Syslog helps you sort through the data overload...
to get the message.
A key tenet of security is that, although prevention is ideal, detection is a must. And early detection is critical. Detecting an attack is like a spotting a fire -- it's easier to put out and damage will be minimal if you catch it at the start.
Syslog for Windows
Kiwi Enterprises' Kiwi Syslog Daemon is one of the most popular tools for enabling syslog on Windows devices. It is a fully functioning syslog server for Windows and can integrate across an entire network.
Logs are the eyes and ears of your network; they capture events and tell you when fire breaks out. Whether it is building a firewall rule set, tuning an IDS or validating which ports should be open on a server, logs are going to give you the information you need to make these critical decisions -- if you are looking.
It's practically impossible to wade through the volumes of log data produced by all your servers and network devices. It is critical to put all events into a single format so they can be analyzed and correlated. Syslog exists in native form for most non-Windows operating systems and devices including Unix, all Unix daemons, routers, firewalls, switches and Web servers. There are free and commercial tools, such as Kiwi Enterprises' Kiwi Syslog Daemon, that allow Windows to integrate with syslog. (See "Syslog for Windows")
With a relatively small investment of time, you can learn how to reap the following benefits of syslog:
- Visibility into network activity.
- Creation of a central repository for host logs, giving you a single location for backing up or analyzing log files.
- Correlation across many diverse systems. Typi-cally, for example, server logs are viewed independent from router logs. With syslog, you can view them all together to better understand what's happening on your network.
- Validation for other devices. A syslog server can make sure other devices are working properly. For example, if a firewall is supposed to be blocking a certain type of traffic that appears in a syslog entry, it means something is not working or is configured incorrectly.
- Detection of attacks by enabling you to quickly spot potentially malicious events.
How to View, Filter and Read Syslog Data to Detect Attacks
Syslog addresses the problem of information overload by breaking down log data into categories that can be easily managed and analyzed. The two most common categories are log source and criticality, which map to syslog message components "Facility," showing what daemon or service originated the message, and "Severity," weighing its importance.
For logs to have value, it is critical to configure syslog properly.
Each facility can be sent to a different location or reviewed by a different group to provide maximum flexibility and checks and balances. (See "Syslog Message Sources") Large enterprises can create multiple syslog servers, each devoted to a particular facility or set of facilities; for smaller organizations, a single syslog server that logs different facilities to separate directories will do nicely.
Kern and other critical services should be reviewed more often than some of the less important services running on a system. However, it is essential to keep messages from all facilities; they might give tips into a system problem or the root cause of a compromise. For example, during one incident, a system had a rootkit, but the company could not figure out how. Since it was a kernel-level rootkit, they focused their energy and effort on the kern and system facility messages. It was not until they examined the mail facility that they noticed an unusual packet size entering the company. Syslog provided the data, but a human had to do the manual correlation.
Some versions of syslog have built-in alerting capabilities; others can incorporate alerting through simple scripting.
As the volume of log data grows, however, tracking down an attack, even using syslog, becomes more difficult. This is where syslog meets up with SIM/SEM tools. These tools do the initial correlation and analysis to narrow down the events, so a human can do root-cause analysis much more quickly.
Severity level is used to determine the message's importance. (See "Syslog Message Severity") You can set the severity level for each device based on the impact a compromise would have on the organization and how quickly someone would need to react. Many organizations make the mistake of not properly defining severity levels and therefore don't get the full benefit of using syslog. For example, if my pager is going off all the time with emergency-level alerts, after a few hours I am just going to ignore it. However, if it is tuned and I hardly ever receive emergency level alerts, I'm going to react immediately when it goes off.
If the severity levels are set correctly, syslog performs basic functions much like a host-based intrusion detection system (HIDS). Using syslog or a contemporary HIDS/IPS product that uses a centralized syslog server, you can address several key security issues:
- Single sensor. If you think of each device as an HIDS sensor, a centralized syslog server can correlate data from multiple hosts.
- Network IDS limitations. One of the problems with a network IDS (NIDS) is that it doesn't see what the host sees. Fragmentation and other TCP-based attacks could trick the NIDS into thinking it is one type of traffic when in reality the host will process it differently. A centralized syslog server will see what the host actually processed. This becomes critical as more traffic is encrypted, since a host logs its information after the data is decrypted.
- Validation for tuning HIPS. As HIDS technology becomes more accurate, a natural evolution is for the software to not just detect, but prevent attacks. However, organizations have no tolerance for HIPS false positives and need a way to validate an attack before blocking it. Syslog can function in that role not only for a single host, but across an entire network.
- Monitoring firewall rule sets. A key principle in tuning a firewall is to adhere to a principle of least privilege, but this can be difficult. Since a syslog server can log information on each individual service, this could be tracked back to each port that is allowed through by the firewall. Now, you can map the firewall rule set back to each host and use syslog to validate whether that port or service really needs access through the firewall.
Best Practices: How to Configure Syslog
Syslog provides insight into a network, but to get the most value out of it, it must be configured correctly. Here are some tips and tricks for maximizing the value of syslog at your organization.
- Compare local and remote logs. If an attacker gains control of a device, he can modify local log files to cover his tracks. However, a central syslog server provides a record of what happened, even if the local logs are altered. You can write a script to automatically compare remote and local files; if they're not identical, chances are the device has been compromised.
- Change the syslog config file. The default syslog configuration file is etc/syslog.conf. Most attackers will look in this file, and if they see that the logs are being written to a remote syslog server, they will try to stop the logging daemon or perform a DoS attack against the centralized server. You can trick them: Create a bait etc/syslog.conf file that is only logging to the local host, then create your real syslog.conf file in a different location and recompile the syslog daemon to point to that new location. The attacker will have no idea that you are logging remotely.
- Employ write-once logging. Since syslog information may be used as evidence in court, you need to ensure its integrity. The simplest solution is copying it to a write-once media, typically a WORM drive.
- Transmit syslog over SSH. Syslog information is sent in plaintext and can still be observed as it is going across the network. To protect your syslog transmission, pipe it over SSH. SSH is a staple on almost every computer system and has minimal overhead. In addition to encrypting all of the information, SSH allows you to add strong authentication and host validation, which increases the reliability of your syslog data.
- Use network time protocol. To correlate data across many systems, it is critical to make sure that they all have a reliable time source. Network time protocol (NTP) is an easy way to make sure every system is using the same time.
Syslog has some important limitations. First, there is no validation. Any system that sends properly formatted syslog messages to the server will be processed and received. An attacker could spoof messages to the syslog server to provide false information. For example, an attacker could report a phony attack to divert your resources away from his real effort.
In addition, since syslog uses UDP, it provides no guarantee of delivery. Therefore, if a system-flooding DoS attack is launched against the syslog server, legitimate messages will be dropped.
An enhanced version, syslog-ng, is based on TCP with authentication. By default, TCP provides guaranteed delivery. Because there is a session setup and tear down, additional information can be exchanged to validate the authenticity of the information and the host that is generating it by adding support through other protocols like IPsec.
Syslog-ng also provides built-in log rotation. Syslog typically loses some messages while the daemon is stopped and restarted as the logs are being rotated out. The challenge is to rotate logs out to archive for compliance, legal evidence and future investigations without losing messages. Syslog-ng will automatically archive files to a designated location.
In addition, while syslog provides basic logging information, it sometimes creates problems when forwarding log messages or providing details of what is happening across a wide range of systems. When forwarding a syslog message, the header of the original device is often overwritten with the details of the forwarding device. For example, if one device forwarded messages from three systems the original information would be lost. In addition, original syslog contains limited information in the header. Syslog-ng helps to solve this by providing additional configurable details on what is happening, where the message was generated and when it occurred.
Syslog-ng is available for most devices and is usually obtained by the vendor. However, since some devices don't support it yet, you can configure your syslog-ng server to support both legacy syslog devices and the newer format.