Security: Measuring Up

Metrics are the key to measuring security. Learn how to gather data and calculate the answers you need.

Here's a formula most security pros will recognize:

Risk = Threat x Vulnerability x Expected Loss

It's useful for expressing the necessity and purpose of security. It's also equally difficult to quantify with meaningful numbers.

How do you numerically express a threat? What is the cost of a vulnerability? How do you calculate expected loss? And, when you multiply these three variables, how do you denote risk in a way that can be translated into an action item?

Security metrics--the measure of security policies, processes and products--is the much-sought-after solution to this conundrum. Security managers look for a magic formula that calculates risk and effectiveness in reducing risk, but the reality is that security metrics aren't that simple.

Measuring security is about using common sense. Managers need to determine what to measure, organize the variables in a way that makes them manageable and meaningful, and build repeatable formulas that show the snapshot status of security and how it changes over time.

We'll define some of the underlying elements required for security metrics and how security managers can use them to quantify different portions of their security programs. We can't provide all the answers, but we'll provide you with a better understanding of the complexity of security metrics and insight into how you can leverage existing data to strengthen performance measurements.

Calculating Asset Value
Enterprises routinely place values on all information assets (hardware, software and data) that are reflected in IT spending. Enterprises expect every hardware and software installation to return an amount that's at least equal to its total cost of ownership (TCO) over its life expectancy.

If we accept that premise, we can assume that the minimum value of all computing assets is the amount of IT spending for a year (salaries, operations and maintenance) plus the depreciation or amortization value of the assets (hardware and software). An ERP system with a TCO of $20 million must return at least that much over its lifespan.

But such calculations need context. That's why we assign quantifiable values to information assets for objective evaluation and comparison. The following are some ways to classify your information assets' value:

Productivity Value. An asset's worth is at least as much as the costs of implementing, maintaining and using it. For a single PC, the minimum information asset value is the cost of the PC plus software, a percentage of IT overhead costs (e.g., help desk) and the user's time (salary). As you'll see later, productivity value is essential for calculating other security metrics.

Revenue Value. For some assets, worth is measured in the value of transactions. If your e-commerce Web server processes $1 million in transactions a day, it's worth $365 million annually.

Revenue value isn't always clear-cut. Supply-chain systems, such as manufacturing equipment and control systems, don't generate revenue, but your revenue generators are useless without their productivity. Their value is often calculated by measuring how much revenue would be lost if those systems were unavailable.

What isn't as easy to calculate is the revenue value of a pure information asset, such as software, music, movies and electronic documents, that can be perfectly replicated. The assets' values are usually much higher than their assigned prices, since they're sold many times over. The value of an "information asset for sale" can be estimated using historical data and/or revenue trend information.

Liquid Financial Assets Value. Those much-vaunted "assets under management" figures associated with financial institutions provide a straightforward way to assess their value. If $1 billion is under management, that amount, plus the productivity value, provides the total value that's being protected. Add to that the revenue values of transaction assets, and you've got the total value of information assets that require protection.

Intellectual Property Value. This is the most difficult asset to value. It's usually seen as "the reason" a company is in business. There are books with elaborate formulas to calculate intellectual property value; however, it may be easiest to consider intellectual property's contribution to a company's market capitalization. This value can be calculated by multiplying the amount of intellectual property captured on systems with the difference between market capitalization and book value.

Calculating Potential Loss
Calculating potential losses is a little different than measuring asset value. Information assets aren't usually "lost" in the traditional sense--they're often still available for use, and partial (or even greater) value may be lost. Some information, such as Social Security numbers, don't have an inherent value; music files that are sold multiple times will have a greater asset value than face value because of their potential revenue. Value is a matter of context.

Because asset value is linked, but not tied, directly to loss, you must consider the type of compromise when evaluating potential losses. There are five distinct types:

Confidentiality breaches occur when high-value information, such as trade secrets or financial data, is read from storage, sniffed from the network by unauthorized parties, or leaked by internal users.

Integrity breaches occur when data is modified in transit or at rest, such as with transactions that are modified to reflect inappropriate quantities or dollar values.

Availability breaches are when information is deleted or made unavailable.

Productivity breaches occur when resources are disabled, such as when an e-mail server is disabled by spam or an e-commerce service is DOS'd by transaction requests.

Liability breaches occur when systems and data are still available, but are being misused. This could be an employee using corporate resources to host a porn site or a hacker storing stolen files on a compromised server.

Determining information asset value enables enterprises to focus on their real security needs and allocate adequate resources. But asset value isn't the only thing that may be lost. It's important to consider incident costs with the lost asset value when evaluating actual and potential losses. Incident costs are the lost asset value that are specifically associated with the incident, including IT productivity, legal and regulatory costs.

The severity of an incident and its losses are calculated by correlating the type of breach with the value type. For instance, if a worm disables your network, you have an availability and productivity breach that affects the asset values of everything on the network--productivity and revenue. The loss equals productivity and revenue values, plus incident costs associated with IT productivity. Legal costs and regulatory fines are also a factor.

A popular method of calculating potential losses is annual loss expectancy (ALE), an estimate for expected loss calculated by multiplying the probability of a particular type of loss by the total loss potential. For example, if there's a 10 percent chance of losing $1 million, the ALE is $100,000.

Obviously, there's a difference between a 1-in-10 chance of large loss and the strong likelihood of a number of incidents throughout the year. One way to address this difference is by treating ALE based on the category: recurring losses or specific security breaches. These distinctions allow you to rate the viability of projections based on historical data. If you know you've suffered an average 100 malware infections a month (recurring), you know your projections have a greater validity. Predicting less-frequent attacks is much more ambiguous, making such calculations more art than science.

Measuring Security Spending
As part of the overall security management program, it's important to measure total spending and spending effectiveness. Measuring enterprise-wide security spending is more difficult than it appears. Security spending is often divided among various business units and departments, as well as being lumped in with network and infrastructure spending. Finding all of the security dollars and separating them from other budget items is a daunting task.

Sort through the IT chaos using activity-based management by classifying different forms of security activity and correlating all spending related to them. An example of this is the "Four Disciplines of Security Management": trust management (policy enforcement, security architecture, certificate management), identity management (user account management, password resets), vulnerability management (configuration hardening, vulnerability remediation) and threat management (monitoring, threat analysis, incident response, forensics). The classification system doesn't matter, as long as you're consistent in your application.

What spending information do you collect for your activity-based analysis? Everything. Spending is broken down into salaries (the percentage of people's time spent on security), hardware and software, and support and maintenance costs.

Staff (people) cost is a matter of identifying wage data for the portion of time anyone spends on security activities. The goal is to identify the percent of time staff members spend on security and calculate it as a percentage of their salaries. This is similar to calculating productivity. After doing this for every individual, the aggregate amount is the total enterprise security spending for personnel.

Allocate these amounts across each of the security activities in the model. The easiest way to do this is to allocate time by percentage for each activity. The percentages can be used to calculate spending for each activity by employee, department or the entire enterprise.

Finally, collect spending for products related to the security activities. The TCO includes product costs, and maintenance and support fees. Dividing TCO by the product's life expectancy is useful to annualize the costs (this is similar to how accountants depreciate and amortize costs over a number of years).

The result is a clear picture of how much an enterprise is spending on security, which can then be broken down by department for comparative purposes. Over time, activity-based analysis will be able to show trends in security spending. And, it's a good method for evaluating security purchases and calculating return on investment (ROI) (see "Positive Returns").

Measuring and Reducing Risk
Traditionally, security managers estimate risk (the probability of something bad happening) as a percentage incorporated into the ALE formula. If the expected loss from a compromise is $100,000, and you estimate that it could happen once every 10 years, the risk is 1 in 10 and the ALE is $10,000.

But which do you pay more attention to: the 10 percent chance of losing $10,000, or the 1 in 100 chance that you'll lose $100,000? Again, risk is often like art--judged by the eye of the beholder.

Defining risk for an entire enterprise is next to impossible. A more manageable approach is breaking it down into small units and types. There are three common forms of risk:

  • Manifest risk is the ratio of malicious events to total events in a computing environment. There are at least four event types: flows, sessions, program commands and transactions. You calculate manifest risk by counting the number of flows and either estimating or using historical data to count the number of malicious events.
  • Inherent risk is the likelihood that system configurations will contribute to a compromise. This is calculated by counting a system's configuration attributes like open ports and services running, and comparing the number of associated vulnerabilities.
  • Contributory risk is a measure of process errors or mistakes made during the normal course of operations and maintenance that contribute to a compromise. Errors associated with user account management procedures and vulnerability management procedures are fertile ground for measuring contributory risk.

Even with these three risk types, you still need to determine the security event/incident frequency. First, you can use historical data to predict the future. If you can identify, for example, the volume of malicious traffic using IDS logs and have already counted the total number of flows, you can project future risk.

Obviously, it will take some time before these numbers are available, and some may never be. But there are other benefits: Even without a number for malicious events, it's possible to measure an environment's exposure.

Exposure is about possibility rather than probability (risk). Exposure measurements can be used as a relative comparison within an environment or across companies. If one can assume that risk is constant for like-sized companies (even if we don't know the number itself), this exposure measure can act as a "risk proxy" to measure the relative difference in risk levels.

Given the elusiveness of quantifiable risk, exposure can provide a means to measure your effectiveness at reducing risk.

None of this is designed to answer all your questions about metrics. But, methodologies outlined for calculating asset value, potential loss, security spending and risk give you a foundation for gathering data and applying it to your own goals and expectations. And, it will bring you closer to assigning values to that security formula and defining what risk means to your enterprise.


About the author
Pete Lindstrom, CISSP, is research director at Spire Security and a contributing editor for our sister publication Information Security magazine.

Note: This article originally appeared on Information Security magazine.

This was first published in February 2005

Dig deeper on Enterprise Risk Management: Metrics and Assessments



Enjoy the benefits of Pro+ membership, learn more and join.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: