How to estimate log generation rates
Are there any calculators to help estimate log generation based on number of devices and best practices?

    Requires Free Membership to View

    SearchSecurity.com members gain immediate and unlimited access to breaking industry news, virus alerts, new hacker threats, highly focused security newsletters, and more -- all at no cost. Join me on SearchSecurity.com today!

    Michael S. Mimoso, Editorial Director

    By submitting your registration information to SearchSecurity.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchSecurity.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

Estimating log generation rates is tricky, and it's difficult to create a credible, generic estimation tool. Many security information and event management (SIM/SIEM) vendors have proprietary Excel-based calculators that they offer to current and prospective clients, but the objectivity of these tools is questionable given their source. I was unable to locate an independent calculator that offers this functionality.

Why is this so difficult? Log generation rates vary significantly based upon the configuration of devices. For example, you and I may both run Microsoft SQL Server databases, but I may have the logging and auditing settings configured to track almost every activity the database performs, while you may have minimal (or no) logging configured. Additionally, I may be in a high-load 24x7 data processing environment, while you may be running a database with low transaction volume. Therefore, it's impossible to provide a meaningful estimate of the log volume generated by a "typical" SQL Server database. Add in hundreds or thousands of other diverse devices, and the problem magnifies in scope quickly.

So how is it possible to develop a meaningful estimate for your environment? There's only one solution: measure your current activity by, for example, setting up a simple syslog server and measuring the volume of traffic it receives. If the systems are similarly configured, you can save time by measuring the logs generated by a representative sample of your organization's devices and extrapolate from there.

More information:

  • Learn how log management reins in security and network device data.
  • See why the PCI Data Security Standard has forced companies to seek log management help.
  • This was first published in February 2009