Tip

Evaluating and tuning an intrusion-detection system

The vast majority of intrusion-detection systems (IDS) available today allow system administrators some degree of flexibility in configuring the sensitivity of the system's detection and reporting algorithms.

    Requires Free Membership to View

This presents a dilemma to diligent administrators: What is the proper balance between providing adequate intrusion detection and the possibility of overwhelming administrators with false reports?

The answer to this question is not an easy one. Indeed, there is no single answer that will apply to every situation or organization, as varying circumstances dictate different security postures. However, if you have a thorough understanding of the metrics used to evaluate these systems, you'll be more capable of determining an appropriate balance for your situation.

There are three main measures of IDS performance:

  • The False Positive Rate is the frequency with which the IDS reports malicious activity in error. These errors are the bane of a security administrator's existence. They're the "nuisance reports" that require investigation but lead to a dead end. The true danger of a high false positive rate lies in the fact that it may cause administrators to ignore the system's output when legitimate alerts are raised. You may also see false positives referred to as "Type I errors," a phrase borrowed from the field of medical research. Generally speaking, increasing the sensitivity of an intrusion-detection system results in a higher false positive rate, while decreasing the sensitivity lowers the false positive rate.

     

  • The False Negative Rate is the frequency with which the IDS fails to raise an alert when malicious activity actually occurs. These are the most dangerous types of errors, as they represent undetected attacks on a system. The corresponding term from medical research is a "Type II error." False negative rates change in an inverse proportion to false positive rates. As the false positive rate increases, the false negative rate decreases and vice-versa.

     

  • The Crossover Error Rate (CER) is often used to provide a baseline measure for comparison of intrusion-detection systems. As the sensitivity of systems may cause the false positive/negative rates to vary, it's critical to have some common measure that may be applied across the board. The CER for a system is determined by adjusting the system's sensitivity until the false positive rate and the false negative rate are equal, as shown in the figure below. You may then evaluate several different IDSs by running them on the same network and measuring the CER for each. If you're interested in achieving a balance between false positives and false negatives, you may then simply select the system with the lowest CER. On the other hand, if detecting every single attack is of the utmost priority, you may still wish to select the system with the lowest false negative rate recognizing that this selection may increase the administrative overhead associated with false positive reports.

Hopefully, these metrics have given you a good idea of how to properly evaluate an IDS. Keep in mind that there is no single solution suitable for every situation. You must carefully analyze your security requirements and determine the appropriate balance of false positive and false negative potential for your environment.

About the author
Mike Chapple, CISSP, currently serves as Chief Information Officer of the Brand Institute, a Miami-based marketing consultancy. He previously worked as an information security researcher for the U.S. National Security Agency. His publishing credits include the TICSA Training Guide from Que Publishing, the CISSP Study Guide from Sybex and the upcoming SANS GSEC Prep Guide from John Wiley. He's also the About.com Guide to Databases.


For more information, visit these resources:


 

This was first published in August 2003

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

Disclaimer: Our Tips Exchange is a forum for you to share technical advice and expertise with your peers and to learn from other enterprise IT professionals. TechTarget provides the infrastructure to facilitate this sharing of information. However, we cannot guarantee the accuracy or validity of the material submitted. You agree that your use of the Ask The Expert services and your reliance on any questions, answers, information or other materials received through this Web site is at your own risk.