Evaluating and tuning an intrusion-detection system

Evaluating and tuning an intrusion-detection system

The vast majority of intrusion-detection systems (IDS) available today allow system administrators some degree of flexibility in configuring the sensitivity of the system's detection and reporting algorithms. This presents a dilemma to diligent administrators: What is the proper balance between providing adequate intrusion detection and the possibility of overwhelming administrators with false reports?

The answer to this question is not an easy one. Indeed, there is no single answer that will apply to every situation or organization, as varying circumstances dictate different security postures. However, if you have a thorough understanding of the metrics used to evaluate these systems, you'll be more capable of determining an appropriate balance for your situation.

There are three main measures of IDS performance:

  • The False Positive Rate is the frequency with which the IDS reports malicious activity in error. These errors are the bane of a security administrator's existence. They're the "nuisance reports" that require investigation but lead to a dead end. The true danger of a high false positive rate lies in the fact that it may cause administrators to ignore the system's output when legitimate alerts are raised. You may also see false positives referred to as "Type I errors," a phrase borrowed from the field of medical research. Generally speaking, increasing the sensitivity of an intrusion-detection system results in a higher false positive

    Requires Free Membership to View

    Login

    SearchSecurity.com members gain immediate and unlimited access to breaking industry news, virus alerts, new hacker threats, highly focused security newsletters, and more -- all at no cost. Join me on SearchSecurity.com today!

    Michael S. Mimoso, Editorial Director

    By submitting your registration information to SearchSecurity.com you agree to receive email communications from TechTarget and TechTarget partners. We encourage you to read our Privacy Policy which contains important disclosures about how we collect and use your registration and other information. If you reside outside of the United States, by submitting this registration information you consent to having your personal data transferred to and processed in the United States. Your use of SearchSecurity.com is governed by our Terms of Use. You may contact us at webmaster@TechTarget.com.

  • rate, while decreasing the sensitivity lowers the false positive rate.

  • The False Negative Rate is the frequency with which the IDS fails to raise an alert when malicious activity actually occurs. These are the most dangerous types of errors, as they represent undetected attacks on a system. The corresponding term from medical research is a "Type II error." False negative rates change in an inverse proportion to false positive rates. As the false positive rate increases, the false negative rate decreases and vice-versa.

  • The Crossover Error Rate (CER) is often used to provide a baseline measure for comparison of intrusion-detection systems. As the sensitivity of systems may cause the false positive/negative rates to vary, it's critical to have some common measure that may be applied across the board. The CER for a system is determined by adjusting the system's sensitivity until the false positive rate and the false negative rate are equal, as shown in the figure below. You may then evaluate several different IDSs by running them on the same network and measuring the CER for each. If you're interested in achieving a balance between false positives and false negatives, you may then simply select the system with the lowest CER. On the other hand, if detecting every single attack is of the utmost priority, you may still wish to select the system with the lowest false negative rate recognizing that this selection may increase the administrative overhead associated with false positive reports.

Hopefully, these metrics have given you a good idea of how to properly evaluate an IDS. Keep in mind that there is no single solution suitable for every situation. You must carefully analyze your security requirements and determine the appropriate balance of false positive and false negative potential for your environment.

About the author
Mike Chapple, CISSP, currently serves as Chief Information Officer of the Brand Institute, a Miami-based marketing consultancy. He previously worked as an information security researcher for the U.S. National Security Agency. His publishing credits include the TICSA Training Guide from Que Publishing, the CISSP Study Guide from Sybex and the upcoming SANS GSEC Prep Guide from John Wiley. He's also the About.com Guide to Databases.

For more information, visit these resources:

This was first published in August 2003

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.

Back to top