The questions about whether the Payment Card Industry's Data Security Standard is working to protect consumer data have resurfaced in the aftermath of high-profile breaches. As a former risk auditor, I've questioned the value of PCI requirements and whether this baseline assessment is working better than unsanctioned alternatives. It is unfair to make this assertion, however, without elaborating on how "working" is defined and what variables contribute to this approach.
The objective of PCI compliance is to reduce fraud by protecting debit and credit card information in retailers' processing environments. There is a decent amount of conjecture on this topic, however. My approach is to take the objective at face value and to evaluate PCI DSS on its ability to reduce risk to the credit card ecosystem.
To determine whether PCI compliance is working, let's assess the data security standard based on its ability to reduce risk in organizations within the constraints of available resources and opportunity costs -- all the things we could have done but didn't once we made our decision. The questions implied by this "PCI economics" approach include the following:
1) Does PCI DSS reduce the likelihood or impact of a credit card data breach from occurring?
2) Are the costs of PCI compliance lower than the financial value at risk?
3) Are there alternatives to PCI compliance that could achieve the same benefit for a lower cost?
In wake of the Target Corp. breach, it is worth considering the broader question: Is PCI DSS "enough" to meet the interests of all parties?
Does PCI DSS reduce risk?
The most important aspect of PCI DSS is its suitability to the task: Does applying the set of required controls make an organization less likely to have an incident or reduce the impact of one? On the surface, it's easy to answer positively, given the breadth and prescriptive nature of the data security standards. But we need to dig deeper.
We can say PCI-compliance was never intended to provide complete protection -- we can even retroactively revoke passed audits -- but we can't escape the expectations.
All organizations (except brand-new ones) start with controls in place prior to implementing PCI DSS. Most enterprises do something even if it isn't required, so we first need to tease out the specific effect that PCI DSS has had on risk. In economic terms, this is defined as the marginal utility of PCI DSS, and here's where things get tricky.
One way to determine whether PCI DSS is reducing risk is to see if incidents of credit card data loss have decreased since PCI controls such as encryption, access control and vulnerability scanning were instituted. The beauty of this outcome-based approach is that it cuts to the heart of the matter -- fraud incidents.* If these numbers are going down, it is reasonable evidence that PCI DSS has reduced risk. But if fraud is increasing, this outcome-based approach isn't fair, because it doesn't account for growth in transaction volume. A better way to determine the marginal utility of PCI DSS is to compare the number of incidents within a PCI-scoped population to a similar one that doesn't fall under PCI DSS constraints. (See the back-of-the-envelope-style calculations in my blog "Is PCI Working?") If the percentage of organizations that experience incidents in the PCI-compliant group is lower than that in the non-PCI-compliant group, then PCI DSS does reduce risk. Unfortunately, it is difficult to find and define that non-PCI control group.
Another way to answer this question is to isolate the new controls implemented due to PCI DSS and evaluate their net effect on reducing risk. According to the Verizon 2014 PCI Compliance Report, "In 2013, companies were compliant with an average of 85.2% of controls." If this was the first year of PCI DSS we could assert that PCI-compliance is responsible for adding that extra ~15% of controls to an environment. (It is not clearly specified that these measures reflect the initial audit, and ultimately, these companies must implement 100% of controls in order to pass.)
Since it is reasonable (albeit pessimistic) for advocates to suggest that at least some portion of the 85% of controls are implemented in preparation for the PCI audit, we must determine for ourselves what portion of the controls were implemented due to PCI-compliance. To gain insight into this, we can look at adoption rates for specific controls across all enterprises, not just PCI-regulated ones. Presumably, these overall numbers will be lower than the PCI-only population; if they are the same or higher, then the argument that PCI-compliance was responsible for the control is a specious one.
PCI requirements contribute significantly to increased adoption of encryption and monitoring, and have a positive impact on privileged accounts and separation, but the other controls in the regulation are already in place in organizations or required by other audits. These extra controls are unlikely to contribute to a reduction in incidents. The challenge with all of the controls is that IT environments are so complex that applying and managing these controls with enough breadth and depth to make a difference is close to impossible. However, it is reasonable to suggest that PCI DSS is reducing risk. But there are still more questions to answer.
Is PCI cost-effective?
The second question is whether PCI-compliance is cost-effective. In order to determine this, we need to look at the cost of implementing the program and compare it to any net benefit. If you don't believe PCI is reducing risk, then this question is moot.
Assuming PCI DSS is reducing risk, the goal is to understand the costs associated with the PCI-compliance program. We evaluate marginal costs in the same way as marginal utility: Only those costs that are associated with the regulation itself should be counted. These costs must be compared to the benefits or the amount of reduced risk. If costs are higher than the benefits, the PCI program isn't working.
The key to measuring the benefits is to look at the anticipated losses due to the additional controls. Calculating losses is, unfortunately, getting easier as more public information becomes available about response and recovery costs, and liability and notification costs, related to high-profile breaches. Of course, estimating the losses associated with a lower amount of future revenue is much harder, but there is little evidence that incidents have long-term impact.
But just looking at the loss potential neglects the uncertainty of future events. The probability that the event may happen must be factored in. Contrary to the popular belief that "everyone is compromised," some discounting must be done to align potential events with periods of time. This is no easy task.
Given the challenges of calculating risk outright, we have to create a practical test (that is not quite as good). In this case, we can perform a break-even analysis. When we spend money on technology, we affirm that the purchase is "worth it." In the technology risk field, this means we've lowered risk by more than we are spending and we can evaluate scenarios that would either support or refute that notion: A $100,000 purchase is worth it when it replaces a 1% chance of losing $10 million, a 10% chance of losing $1 million and so on, up to a 100% chance of losing $100,000.
It is important to remember that these marginal costs address marginal utility, which means we isolate the PCI-compliance spending and compare it to the incremental risk that is reduced by this activity. As with question No. 1, it is possible to conclude that PCI compliance is not only reducing risk, but it is also cost-effective. Ultimately, I have a hard time believing that all the money spent on PCI compliance is actually reducing risk by, at least, the same amount.
Does PCI provide the greatest return?
If you thought questions No.1 and No. 2 were difficult, then question No. 3 is even harder. It brings up the possibility of opportunity cost: that some alternative approach, which isn't sanctioned by PCI DSS, could actually reduce risk by more than a standardized control.
Say, for example, an organization is better off migrating from a signature-based antivirus technology to an endpoint container. At this point in the evaluation process, the sky's the limit when it comes to reviewing and evaluating technology risk options for protecting credit cards. The only difficulty is that the PCI Security Standards Council may elect to levy a fine on companies that are not compliant. This, too, must be factored into the costs of the alternative control.
A small but growing contingent of organizations has decided that it is better off accepting the possibility of being fined than paying the up-front costs for control mechanisms that it believes are ineffective to address PCI requirements. Question No. 3 is also the point where organizations that answered "yes" to questions No. 1 and No. 2 are unlikely to budge from their position, and those that answered "no" have better things to do -- like protect their environment (sorry, I couldn't help myself).
Is PCI DSS enough? Regardless of the answers to the first three questions, the challenge of determining whether PCI compliance is enough is huge, considering that the public-facing test revolves around a breach incident. With the public expectation of perfection, we are really in a bind. Any disclosed loss of credit card information due to a breach is too much. We can say PCI compliance was never intended to provide complete protection -- we can even retroactively revoke passed audits -- but we can't escape the expectations.
Different mindset needed
The true failure of PCI DSS is not in its attempt to attain perfection; it is the absence of proper cost-benefit analysis (instead of waving a magic wand). It appears that the PCI Security Standards Council is perfectly fine with using this same form of magic when determining the need for newer controls, such as chip-and-PIN cards.
Alternatively, the council has the authority, and the affected parties could gather the applicable information, to create metrics and measures that would help determine appropriate levels of protection. This information should relate to the value of the transactions, the volume of activity, the outcomes of controls, the number of control failures and the magnitude of losses.
At some point, with the proper data, the credit card world could create a system in which organizations are responsible for their own security and -- based on the level of risk these companies are willing to accept -- set aside funds, or contribute to a reimbursement account, or purchase insurance, or create some other way to address the anticipated losses.
It seems more likely that the PCI Security Standards Council will attempt to strengthen the prescriptive, controlling nature of its requirements, still aiming for that perfection it cannot achieve. And that would be a shame.
*While credit card information disclosure is the focus of the risk described in this article, it is worth noting that the "truer" risk is associated with the actual fraudulent usage of this information. Only a fraction of disclosed credit cards are successfully used for fraud.
Pete Lindstrom is principal and vice president of research for Spire Security. He has held similar positions at Burton Group and Hurwitz Group. Lindstrom has also worked as a security architect for Wyeth Pharmaceuticals, and as an IT auditor for Coopers and Lybrand and GMAC Mortgage. Contact him via email at PeteLind@spiresecurity.com, on Twitter @SpireSec, or on his website, www.spiresecurity.com.
- Meet PCI DSS Requirements –HackerOne