alphaspirit - Fotolia
Risk management metrics on third parties, such as the number of vendors that have access to company information, are usually an eye opener to boardrooms, says Terri Curran, senior security consultant at CGI Group Inc.
Information security metrics should concentrate on three areas -- risk management, compliance and innovation efforts. But trying to do too much at the start of a metrics program is a common misstep, acknowledges Curran, a 40-year InfoSec veteran, who performed the CISO function for 19 years at The Gillette Company and for seven years at Bose Corp.
Curran holds Certified Information Privacy Professional (CIPP), Certified Information Security Manager (CISM), Certified Information Systems Security Professional (CISSP), Certified Protection Professional (CPP) and Certified Risk and Information Systems Control (CRISC) certifications. She earned both a bachelor’s and master’s degree in security management from American Public University System, and she is currently a doctoral candidate at Nova Southeastern University, where she is specializing in InfoSec training standardization.
She served on the Computer Security Industry (CSI) advisory board from 2007 to 2014. (CSI was absorbed by UBM in 2011.) Marcus Ranum caught up with Curran to discuss how to make metrics work for a broader audience.
Marcus Ranum: Let’s talk about metrics -- the most boring topic in computer security, behind configuration management … and everything else. When the topic comes up, I often experience this dialog:
“You should keep metrics.”
“What metrics should I keep?”
“I don’t know -- it’s specific to your enterprise.”
“Isn’t there a ‘top 20’ metrics list?”
How do you make the connection between theoretical and operational usefulness? How do you respond when someone asks about the top 20?
Terri Curran: I love talking about InfoSec metrics: I absolutely believe that metrics discussions can explode into ideas and innovation. I also believe that metrics are boring because a lot of them aren’t relevant to today’s executive boards. I’ve been in boardroom meetings where as soon as the CISO’s metrics presentation flashed on screen, eyes rolled heavenward and email was surreptitiously checked.
Don’t get me wrong: Metrics are important. But metrics, like security itself, have to evolve with the times. Is there a top 20? Sure, there can be for any organization.
Ranum: In my experience, the most useful metrics are often ones that quantify time spent on stuff. I’m never sure if that’s because establishing such metrics entails doing some business process analysis, or if it’s just that time metrics tend to be a useful yardstick for effort and expense. What’s your experience?
Curran: Based on years of metrics mistakes and successes, I think the most useful, current metrics illustrate risk management, compliance and innovation efforts. Some of these metrics are absolutely time-based, some are not (particularly in the ‘innovation’ category). I think you need a mix of time-based, results-based and forward-looking metrics to explain your InfoSec posture and avoid the rolling eyes in the boardroom.
The old SMART (specific, measurable, actionable, relevant, timely) yardstick can be used very effectively to start the metrics definition process in any of the three categories. In addition to being SMART, metrics must be easy to manage or track. They need to be multipurpose and multidimensional.
Metrics also need to be reported on by the people who do the work on them. If the IT or data center people are doing all the malware remediation and patching, they should be the ones reporting on that (proudly) as part of their role in the security program. The CISO shouldn’t require them to provide monthly status reports that roll up into her report. Let the IT folks take credit for their hard work, and let the CISO innovate some new metrics. At the end of the year, the IT leader can provide a summary that is included in the CISO’s annual security report with proper acknowledgement. Other business units can do the same with their InfoSec-related metrics to illustrate their commitment and support for the InfoSec program.
Of course, the number of phishing and malware attempts, patches applied -- all the technology-based metrics are great. But I’d like to think of risk-management metrics in broader context. For example, a great risk-management metric might be the number of third parties with access to company information, the number of third-party risk assessments conducted, and number of third parties protecting information properly. Many companies still haven’t considered third-party risk assessment as part of the metrics portfolio, and executives seem to be surprised about the number of vendors and others that have access to their information.
Another risk-management metric might be the number of proactive activities taken to stay current with laws, rules and regulations (webinars, updates from advocacy groups or legal papers). Some of my favorite metrics in this area are outreach activities. If you’re in regular contact with your local law enforcement and public safety officials, it’s a great physical, security, risk management and innovation metric. My point here is that metrics can be people-centric as opposed to technology-centric.
Compliance metrics are pretty straightforward based on the external contractual and regulatory compliance requirements of the organization. PCI DSS, NIST 800-53—lots of requirements provide great metrics as part of execution. Measuring internal compliance to InfoSec policies is easier now because of monitoring technologies like data loss protection that are technology-centric. Of high interest to me is keeping track of the number of external audits that are planned or in process, and the hours and resources needed to support them. That’s an old metric but still very valid and valuable.
Lately, my favorites are innovation metrics. Physical and InfoSec programs need to grow and evolve. We can use metrics to generate and illustrate forward-thinking activities and ideas.
Not all innovation ideas will become reality -- but that’s a metric in itself, isn’t it? On a monthly basis, for example, the security team can have an innovation discussion and come up with ideas that are of benefit from a professional and personal perspective. I’ve seen some really wonderful things come out of innovation discussions that breed even better metrics. Here are a few examples: having a ‘shred day’ where employees can bring documents in for safe disposal—that illustrates good security awareness, generates metrics on the number of employees participating and even the total number of pounds shredded (good corporate social responsibility). Or offer training sessions on free malware software and track employee participation. I’ve heard anecdotally of companies that create security-awareness presentations for employees to take home and share with their kids. I also like to show ‘number of business unit meetings held to discuss security issues.’ The best thing about innovation metrics is that if 10 are generated in a month, but only three come to reality, so what? You’re still showing innovation.
Ranum: It seems to me that a lot of enterprises are getting shellacked by malware and basic phishing attacks. When I see that, my assumption is usually that they don’t actually understand how badly it’s hurting them because they need metrics. What are some of the things you’d measure in order to explain to senior management the impact of malware and end-user computing practices on the organization?
Curran: Some organizations have an appetite for social engineering and phishing tests, and these are great metrics to measure employee awareness—and [they] present great input for training programs as well. I think we need more illustration for management to benchmark how their organization compares to others in their industry. Metrics can be used to show maturity gaps (or, even better, good posture). I wish there was an external monthly report or digest, which I could show senior leaders, that indicates by industry, or sub-industry, how many malware and phishing attacks were reported in a given month, and how the company stacks up against those reports. That would be a huge win. If you know of a resource like that, let me know, please!
Ranum: I used to be dismissive of risk management because it often seems to be a case of ‘garbage in, garbage out.’ But when I started digging into metrics, I realized that the only way that you can actually make any sense of this stuff—and reduce the garbage level -- is to start measuring things. Where do you start? If you were building a metrics program from scratch, what are the first steps?
Curran: I developed a rudimentary worksheet for metrics development that might explain my approach to identifying ‘value-add’ metrics. It also can be used as a resource-planning tool or as input to a RACI [responsible, accountable, consulted and informed] model.
The metric itself needs to be measured as either qualitative (narrative in this section) or quantitative (percentages or other numeric value). The most important part of this worksheet is: What is the question being answered? If there’s no question that’s of interest, then the metric really isn’t worth chasing. Data sources could be tools used or surveys issued; and the last two columns are pretty straightforward.
The first place to start is the compliance requirements. Then I’d tackle risk management metrics and, finally, innovation metrics. It’s a maturity curve that can build out gradually. And don’t forget to keep metrics on metrics: We created and reported on five new metrics last month.
Ranum: What’s the biggest mistake people make when they are starting a metrics program, and how should they avoid it?
Curran: I have been guilty of probing too deeply into InfoSec metrics and looking for a complete portfolio to start, trying to add too many items at once. I’ve learned that the best metrics show the value of the InfoSec program to the company as well as to external regulators and auditors. Trying to accomplish too many metrics for the sake of metrics is a failed effort before even starting.
Ranum: OK, we talked about the top 20. I’m putting you on the spot: What are your current faves?
Curran: Well, how about five of each to start? Some of them we’ve already talked about, but here are metrics I’d be looking for if I were concerned about protecting my company’s information and physical assets:
- Third-party risk assessments performed
- Number of internal and external audit findings issued (with current status, in process, completed and so on)
- High-level data leak prevention statistics (could also be considered a compliance metric)
- Benchmarking to similar industries based on analyst, vendor reports
- Number of vendors with access to restricted, confidential or regulated information (with trending)
- Patches applied and all the usual IT-related topics
- Policy exceptions requested and granted
- Schedule of compliance activities with trending (increase is likely)
- Number of contracts reviewed for security or privacy concerns (could also be considered risk management)
- Number of regulatory or contractual research hours conducted to stay on top of upcoming changes
- New ideas generated (with brief explanation)
- Ideas approved for moving ahead
- Outreach meetings initiated (business units)
- Outreach meetings initiated (external agencies, regulators, interest groups; could also be considered in risk-management category)
- Personal and professional development outreach (certification guidance, new certifications acquired)
Ranum: I often hear computer security people say that security doesn’t know how to talk to executive management or business units. What do you say to that?
Curran: That’s a great question. My first response is this: Don’t talk. Listen. Yes, I know, this sounds obvious and a little snarky. But we often don’t listen enough to our colleagues in the business side of the organization.
So now to the serious answer: Talk less about technology and more about people-centric risk management. Remember that people want their organizations to be safe and successful and [they] want to feel they are part of this effort. Appeal to their protective instincts without being alarmist. Gain their confidence by showing sincere interest in how their business process integrates with security. These are great ways to get conversations flowing and build their trust.
About the author:
Marcus J. Ranum, the chief of security at Tenable Network Security Inc., is a world-renowned expert on security system design and implementation. He is the inventor of the first commercial bastion host firewall.