Enterprises have adopted security information management systems (SIMs) for their value in correlating, reporting, and alerting on network security. By feeding firewalls, intrusion detection and prevention, and vulnerability analysis into a common platform, network and security managers have a valuable window, giving greater visibility and helping to clear out the noise. Despite their name, though, SIMs can be used for more than network security monitoring. In many cases, the same tools can bring value to application managers if they’re used correctly. With attacks targeting the application layer, SIMs can help find security problems in enterprise applications that otherwise might get missed. But SIMs can do more than identify security threats: Any hard-to-find event or application performance issue can show up through careful analysis.
We’ll walk through the four steps application managers need to integrate applications into enterprise security information management systems and begin analyzing, reporting and alerting.
Feeding application data into a SIM
Before you begin using a security information management system, you’ve got to start feeding it your application data. In the world of networking and security, this is easy because network and security devices universally support SYSLOG, a way to ship log data over the network to a central point. SIMs love to get data via SYSLOG, so that’s always your first choice if your application supports it. You’ll have to work a bit harder for the rest of them.
For enterprise applications that send logs to Windows Event Log (Microsoft Exchange is the most common candidate here), getting those into a SIM will be easy. Most SIMs already have a simple strategy to connect to Windows Event Logs, either using a SYSLOG converter that sends Windows logs out via SYSLOG, or through some native tool that pulls logs directly from Windows.
With attacks targeting the application layer, SIMs can help find secrity problems in enterprise applications that otherwise might get missed.
Applications that write standard log files or send their logs to databases will pose a greater challenge. Some SIMs have already dealt with this problem out of the box and have tools or daemons that will monitor databases for changes or watch log files as they grow. If your SIM doesn’t do that, you’ll need to figure out some way to get those logs sent over. One approach is to wait for hourly or even daily intervals and use a batch procedure to copy the logs for the last time interval over to the SIM all at once. If your goals in using the SIM are more focused on reporting, correlation and forensics, then having a delay of an hour for logs to be sent over may be fine.
When you start thinking about what logs to send to the SIM, make sure send everything, but some logs, such as back-end database logs, may be more than your SIM can handle. In general, you should start from the edge of the network and work your way backwards. Try and capture the full stack that represents the application from beginning to end. For example, if you have load balancers in front of an application, you’ll definitely want those logs. This is doubly true if you are doing source network address translation (NAT), a common strategy in load balancers for application servers because of the simplified deployment model. Without logs from the load balancers, you won’t be able to track transactions back to the true originating IP address or identify some types of denial-of-service (DoS) attacks. Keep going, including your Web front ends, application servers and database servers. If you’re adding logging for email, make sure you include the whole set of products: antispam/antivirus gateway, email relay, and finally, messaging server pieces.
As you work in toward the back end of the application, keep in mind the potential goals of SIM analysis: correlation and analysis, alerting, reporting, and forensics. If the logs don’t advance one of those goals, they aren’t going to help you much and will just clog up your SIM.
Parse, normalize and store application data
The difference between a SIM and a log storage system is in the ability of the SIM to “understand” the logs you send it, a process generally called parsing or normalization. If you expect to get anything useful out of your SIM, you’ll need to make sure it is able to interpret the logs, collect information from various fields, and generate reports, calculate rates, and correlate information across log entries.
In many SIMs, the parsing and normalizing are dark magic (OK, that’s a technical term) reserved to the SIM vendor’s engineering team. In that case, you’ll have to provide them with a sample of your log files so they can add the necessary logic to their product. You may also need to add fields to the SIM’s database, a task that can either be easy or next to impossible, depending on how flexible your SIM is.
At this stage, be prepared to give up --if your SIM vendor says you won’t be able to do something useful with these logs, don’t try and force a round peg in a square hole. Even if your SIM can’t parse and normalize your application log files, you can still get useful information from the logs. Unparsed log files, though, won’t give you good reports, statistics or analysis. For example, many SIMs will treat unparsed log entries as blobs of text, which would let you raise an alert on certain specific entries, such as fatal errors or security alerts. That’s not giving you the full value of a SIM, but alerting combined with some forensics and log search capabilities can make it worth your time to go down this path.
Don’t look to your SIM for long-term log archiving if that’s going to mean keeping two (or more) copies of everything in two different systems.
If you’re parsing the logs yourself or advising your SIM vendor, you’ll need to figure out which fields to pull out of each log entry. Focus on fields that will make sense for reporting, alerting, and correlation. For example, if you’re pushing email application logs over, you’ll want to try and track fields such as message ID, envelope-from, envelope-to, subject, and date as a bare minimum. When adding more complex applications, such as ERP systems, you’ll have to pick and choose fields from a fairly complex environment. Parse fields that will help you trace a transaction from the originating IP all the way back to the application. That means starting at the front end and moving back one level at a time, always looking for some way to link entries at one layer to the next. For example, if you have a load balancer changing IP addresses, you’ll want to capture the original IP address and the changed IP address. Then, you can link Web server logs with the changed IP address to the original IP address by going through the load balancer logs.
Depending on your log management strategy, you may want to set a shorter expiration date for application-layer logs than other logs you’re collecting. Most application stacks are going to have their own log management systems, making the SIM a duplicate of what’s already being collected elsewhere. In that case, keep what you need for your reporting, alerting and forensics requirements, but don’t look to your SIM for long-term log archiving if that’s going to mean keeping two (or more) copies of everything in two different systems. Most SIMs have built-in reporting tools that will provide some trending information, but unless there is specific report for application-type logs, you’re not going to get much useful trend data over the long term. That means there’s no point in saving three years worth of application logs if you never go back more than three months in forensics.
Writing business rules and alerts
A SIM will never know what you want to do with your logs unless you tell it. All SIMs are rule-based, which means rules are going to be the best way to add value to the task of looking at application logs with your SIM.
Writing business rules isn’t the most fun part of this exercise, but it is the one that leverages the power of the SIM best. When you begin feeding application data to the SIM, you’ll want to focus on two fronts: reusing existing rules and writing new ones to leverage new types of data.
If your SIM has parsed and normalized log files properly, then some of your existing rules will probably apply. For example, a login failure on a database server will be caught by the same rules you have looking for login failures on Web servers or firewalls. This is what you want—you want your application log messages to look as close to existing log messages as you can, so any business logic you’ve put into your SIM will apply to new applications.
However, you will probably have other rules that are specific to your applications. Most SIMs can watch for regular expressions in individual log entries, patterns and rates of one type of log entry, and correlation between two (or more) entries. Application logging can take advantage of each of these features.
For example, the MySQL database will log “slow queries,” ones that take more than ten seconds to run. If you get five of these a day, or even 50, that may be normal, but if the rate of these suddenly jumps up, you want to know about it. This is an ideal job for a SIM, which can dig out slow queries from all the other logged messages and let you know when the rate goes above a threshold. Or you may have certain queries that you know will be “slow,” but want to know about any others. That’s a good use for the rule-based engine in a SIM, because the engine should be able to correlate different log messages together to catch the interesting ones.
Business rules can be simple if you want to just look for regular expressions. For example, you may want to be alerted whenever certain banned IP addresses try to connect to the application, or if a user with a disabled account tries to log in. And if you want to identify application-layer attacks such as SQL injection, for example, it’s easy to write a handful of rules that look for sequences such as “xp_” in traffic going towards a normal Web server.
As you’re writing business rules, don’t forget the value of combining sources of information from outside the application with your application logs. For example, you may already have IDS/IPS events being sent to the SIM. Those can be correlated with logging to filter out problems that have already been solved at the network layer, or to help identify more information about suspicious transactions.
The last step of real-time log analysis is alerting: telling someone or something about whatever you’ve found in the logs. Each deployment will be different, but if your SIM is already installed, you’ll want to try and fit smoothly into the existing alerting strategy. However, you may find network and security teams have a different way of looking at alerts since they are more focused on real-time issues, (“something bad is happening right now!”) while application logs are often more useful seen over a longer period of time, such as a day or a week (“here are all the interesting slow queries from last week”). Don’t be afraid to point this out to the team managing the SIM so they don’t get the wrong idea about what’s most important to you.
Using SIM to handle reporting, archiving and forensics
Security information management systems can help you with reporting, forensics (looking backward at logs to understand the root cause of problems), and archiving of your application logs. If your application is long-standing, you may already have strategies for all three of these activities. Just because you’re adding a SIM into the mix doesn’t mean you have to change your strategy for longterm log retention. With compliance regimes requiring three to seven years of logs, SIMs and their high-speed databases can be an expensive alternative to a simple log server.
Regardless, whether you’re using the SIM for archiving and forensics, you want to filter logs going to the SIM. For example, many applications can write detailed debugging information as well as normal transaction information in their log files. That debugging information might be useful to the application manager, but not so useful to the SIM. You can filter either by ignoring some events when they arrive at the SIM or by filtering them out before sending from the application to the SIM. Which technique you use depends on your application tools and your SIM, but the best approach, if your SIM supports it, is to filter out at the SIM. This way, you can easily increase the types and level of logs you save at the SIM without having to go back to the application manager and ask them to change anything.
Reporting can be a mixed bag with SIMs when trying to add application logs into the mix. If you are generating your own alert data based on correlation and analysis of logs, then reports on your alerts will be invaluable. However, don’t be disappointed with stock reports. The off-the-shelf SIM reports are likely to be aimed more at security-type events, and may not be adaptable to the kinds of things you want to summarize and count in your application logs.
Adding application log data to your enterprise SIM can bring a wealth of information and give you new insight into what is going right, and wrong, with your enterprise applications. However, SIMs are focused on security and the path to non-security data from application logs can be arduous, involving the SIM vendor and work on your part. A successful integration will take time, but will increase your security and application awareness all at the same time.
Joel Snyder is a senior partner with consulting firm Opus One in Tucson, Ariz. He has worked in IT for more than 25 years. Send comments on this article to firstname.lastname@example.org.