Auditing is a core component to compliance and security programs, and a generally accepted practice by IT operations. Relational databases were the first enterprise application to embed auditing as a native platform feature. Yet part of this heritage is the stigma associated with auditing. Vendors provided the basic function, they failed to provide the performance and ease of management demanded by database administrators (DBAs), who remain loathe to enable the feature, and remain one of the last holdouts for accepting the use of database audit trails.
The reasons for their distaste of native audit are fully justified: from performance and data management nightmares, to the fact that these audit trails were not originally designed for the tasks they are being applied to. Regardless, regulatory compliance demands accurate and complete records of transactions, and relational database audit trails produce just that.
Security and compliance drive the use of database auditing, and these log files provide a unique perspective that needs to be captured. Database vendors are building better auditing features into their products, minimizing historic impediments, but careful deployment is required to avoid these known issues.
WHAT IS DATABASE AUDITING
Database auditing is the examination of audit or transaction logs for the purpose of tracking changes with data or database structure. Databases can be set to capture alterations to data and metadata, along with modifications to the database system storing the data. A typical audit record will include the database operation performed, the data values that were changed, and who performed the operation, along with a number of other attributes. Auditing capabilities are built into all relational database platforms, and ensure these logs have the same degree of accuracy and integrity as the data being stored. Further, the audit trail is able to group a series of statements together into logical transactions, providing a business context that makes it a preferred choice for forensic analysis of business processes.
There are some limitations with auditing because data access statements, commonly referred to as SELECT statements, are not collected. Additionally, native database audits seldom capture the original query or variables passed by the user, rather they record a synthesized view of the event. The logs capture data values, both the before after, changes were made. That makes audit trails much more useful for detecting what was changed, rather than what was accessed.
For forensic examination of database activity and state, audit provides an accurate view of events. For review of SELECT statements--people trying to view data--the native platform either lacks the ability to collect these statements, or does so by enabling advanced options that incur a significant performance penalty. It's for this reason that, when used for security, audit trails are most useful for when looking for insider misuse--rather than external attacks. There are easier and more efficient ways of cataloging SELECT statements (e.g., failed logins, attempts to view credit card information) and why organizations choose to supplement native database auditing with other data collection sources. Regardless, built-in database auditing features produce the core information for transaction verification and regulatory controls.
IDENTIFYING DRIVERS FOR DATABASE AUDITING
Before we delve into some of the specific tools to create and collect database audits, let's take a closer look at why these features are being put to use. Your organization may have one, or perhaps all, of the following requirements:
Compliance: Regulatory controls, such as change management, business process verification, system failures or security events material to accuracy and consistency. That's why database auditing is critical for fraud detection, as well as regulatory compliance requirements related to Sarbanes-Oxley. Change management, transaction history, reporting of ad-hoc events are all the types of things normally required for regulatory controls. Supplementary controls for PCI-DSS are relevant as well, with system privilege changes, or any administrative activity and alterations of system functions needing careful scrutiny.
Keep in mind that database auditing is not specifically listed as a requirement in any compliance mandate, including SOX, PCI-DSS, HIPAA, FERPA or any of the state privacy regulations. In practice, audit fills an essential role by providing an accurate and concise history of business processes, data usage and administrative tasks--all necessary elements for policy enforcement. As databases back all enterprise applications, and are the sole authority on data state - the net results of business processing - they are the most logical and effective place to implement controls. As such, most audit requirements that center on tracking a specific set of users, objects, or data elements require database auditing.
Security: Data security and privacy, over and above what is required for the regulations listed above, is the other major driver for database auditing. Capturing failed logins, failed queries and misuses of administrative functions is a way of detecting system probing. Monitoring the add-on of views that can expose data, public permissions to use system functions, and privilege changes that provide administrative capabilities to regular users, are all common use cases. Forensic analysis of who, what, when and where attributes provide fair indication if database usage was legitimate or probable attack. In the event of database tampering, the audit trail provides enough information to determine what changes and help understand required corrective action. The audit trail is commonly used to feed security tools such as SIEM and log management for correlation and security event notification.
Operations: Database auditing was originally designed to help database administrators examine database activity so they had a better understanding of how to allocate resources and what queries needed tuning. While there are better tools to accomplish these tasks today, auditing still provides failure analysis and business process analysis to ensure reliability of database. Because let's be perfectly honest, when a catastrophic failure occurs and you are asking "What just happened?" having an audit trail is a pretty handy tool for recovery.
DATABASE AUDITING TOOLS AND APPLICATIONS
There are four basic platforms used for creating, collecting and analyzing database audits: the native database platform; system information and event management and its cousin log management; database activity monitoring; and database audit platforms.
- Native Audit: Refers to the use of native database auditing for data capture, but use of the database system itself to store, sort, filter and reporting of events. IBM, Microsoft, Oracle and Sybase all offer slightly different variations, but capture essentially the same information. Data is typically stored within the database, but can be exported into flat text files or sent as an XML data feed to other applications. Using the native features saves on the costs associated with acquiring, deploying and managing a dedicated auditing tool, but suffers from additional performance overhead on the database, limited management features over and above basic collection and storage, and requires manual administration. Native auditing occurs within the database, and is used just for the analysis of the databases housed within a single installation.
- SIEM and log management: Security information and event management, and similar log management tools are capable of collecting audit files, but offer many more features that the native database tools. Keep in mind that these tools pull the native audit trails from the database, but a dedicated server is used for storage and processing, relieving much of the burden from the database. They also collect information from network devices, operating systems, firewalls and applications in addition to database audit logs. SIEM and log management offer integrated reporting, data collection, heterogenous database support, aggregation and compression are all advantages over native database capabilities. Log management systems, as represented by companies such as LogLogic and Splunk, are specifically designed to accommodate vast amounts of data, and focus more on management and reporting. SIEM, represented by companies such as ArcSight and RSA the Security Division of EMC, are designed more for analysis, with near real time policy scrutiny, greater depth in event correlation and alerting. Still, differentiation between SIEM and log management will soon be a thing of the past as most vendors offer both platforms, albeit not fully integrated.
- DAM: Database Activity Monitoring platforms are designed to monitor database activity for threats and enforce compliance controls. Vendors such as Application Security, Fortinet, IBM, Netezza and Oracle offer heterogenous capture of database events. Most offer more than one way to capture information, collecting queries from the network, from the operating system under the database, as well as the database audit logs. DAM tools are specifically designed for very fast retrieval of data and real time policy enforcement. Like SIEM tools, they can collect data from heterogenous databases, multiple data sources, and are designed for analysis and alerting. Unlike SIEM, and as they are specifically tailored to databases, DAM systems are much more focused on database analysis at the application level rather than the network or system level. DAM products also offer advanced features such as activity blocking, virtual patching, filtering and assessment capabilities in addition to forensic activity analysis.
- Database Audit Platforms: Some of the database vendors offer specialized databases that resemble log management servers. These are comprised of a dedicated platform that stores log files captured from the native database auditing, collecting multiple database logs into a central location. Some of these platforms offer heterogenous database log file collectors. Reporting, forensic analysis, aggregation of log files into a common format, and secure storage are common benefits. They do not offer the multidata sources or fine-grained analysis of DAM, the correlation and analysis capabilities of SIEM, nor log management's ease of use. But for IT operations focused on database auditing, this is a cost effective way to produce security reports and store forensic security data.
DATABASE AUDITING SELECTION PROCESS
To assist with your selection process for database auditing, you will want to consider the following attributes of each platform type, and each vendor's solution. In the order of importance:
- Data Sources: The primary source of information described in this article is database audit logs, which are created by the database engine. However audit logs vary from database to database, and in some cases there are multiple sources of information that are lumped under the same term: audit log. Further, some platforms can create an activity log of user actions against the database. While this later variation is not as accurate as what is created by the native platform, it does provide all SELECT statements and better performance to boot. You need to carefully inspect what data is available to you from the different data sources, and see if this is enough information to fulfill your security, operations and compliance requirements.
- Compliance: As compliance with industry and government regulations is the key driver for adopting database auditing solutions, review the policies and reports the vendor provides with the product. These reports help you meet compliance mandates quickly, and reduce your costs in customization.
- Deployment: The single biggest complaint by customers for all of the solutions described is the pain of deployment. Installation, configuration, policy management, reducing false positives, customizing reports or management of the data are also issues. It is for this reason you will need to dedicate resources to evaluate tools in a head-to-head comparison. Further, deployment against one or two databases is insufficient: plan on performing some scalability tests across many databases to simulate real world conditions. While this creates a burden for the proof-of concept process, it pays for itself in the long term as UI issues, policy management and poor architectural choices by the vendors only become apparent in real world test cases.
- Performance: This has less to do with the vendor platforms, and more to do with tuning the data auditing options of the database itself. There are many variations and options, and the performance of native audit varies radically, so run some tests. You also need to balance what data you would like to collect with what you need, and look for ways to meet your requirements with the least number of policies possible as more policies means more overhead on all systems.
- Integration: You will need to verify the vendors you review offer integration with workflow, trouble-ticketing, system and policy management products you have in house.
Audit logs contain a lot of useful information helpful to auditors, security professionals and DBAs alike, but they impact performance. Any conversation about the wonderful things that database auditing can provide needs to be tempered by understanding the added burden. Auditing incurs a performance penalty, and depending upon how you implement it, that penalty can be severe. But these issues can be mitigated, and for some business problems, database audit logs are essential to compliance and security analysis.
With the exception of native database auditing--which piggy-backs on top of the database resources--all the tools we have described deploy as a standalone appliance or software. All offer central policy and data administration, reporting, and provide data aggregation capabilities. SIEM, log management and database activity monitoring vendors provide a hierarchal deployment model for scalability, where multiple servers or installations are distributed across large IT organizations to help with processing and storage requirements.
Aggregating data helps ease management and reporting on the enormous volumes of data being collected. Further, information collection into a central server guards transaction logs from tampering.
Which approach is right for you comes down to your requirements, the business problems you need to solve, and (unfortunately) the time and money you can dedicate to the problem. The good news is there is a wealth of options available, from having your DBA turning on the native audit of your database to capture the basics, to products aggregating data across thousands of devices.
Adrian Lane is CTO of consultancy Securosis. Send your comments on this article to firstname.lastname@example.org.