In a disaster, all focus is -- naturally -- on getting critical business processes back up and running. Whether the disaster is natural or manmade, it's all about recovering business operations as fast as possible, getting employees back to work, and avoiding costly downtime.
In this scenario, information security is often far down on the list of considerations, experts say. But companies that overlook data protection provisions in their disaster recovery/business continuity plans risk winding up with a double whammy: a security breach on top of a recovery situation. Imagine having to issue breach notification letters in the midst of recovering from a hurricane or other disaster. After all, compliance requirements aren't lifted in an emergency.
"You need to get folks access to the data if they need it, but you also need to prevent unauthorized access," says Ed Moyle, a manager with CTG's information security solutions practice and a founding partner of consultancy SecurityCurve. "That's where a lot of organizations fall down."
Disaster recovery/business continuity plans must ensure that an organization's information security policies are maintained in a recovery situation, security practitioners and others say. That means making sure the recovery site has proper security, including updated antivirus and firewall protection. It also means conducting proper due diligence of any disaster recovery provider and taking proper precautions in a shared recovery facility. Transmission of data for backup purposes must also be secured.
"What you're doing to secure a disaster recovery site has to be every bit as good as what you're doing in your primary site," says Brian Engle, director of information security at Temple-Inland, a manufacturing firm based in Austin, Texas. "If you end up in a disaster recovery situation, it could be long term, maybe six months…Can you be comfortable with the decisions you make in choosing the facilities and the protections for that length of time?"
SECURITY LEFT OUT
Organizations often don't think about how the security controls they have during routine operation might fare in the event of downtime, Moyle says.
"For example, if you have a security program built around the idea of keeping physical access to things like servers locked down, you may not be able to enforce that to the same degree in an emergency scenario as you could during normal business," he says. "You want to make sure security controls continue to function during a downtime scenario."
Some companies assign disaster recovery planning responsibilities to their security groups, but others focus on databases, servers and networks rather than security reviews in their planning, says William Hughes, director, consulting services BC/DR Center of Excellence at SunGard Availability Services. "They're not as involved as they should be," he says of security teams.
Organizations typically consider disaster recovery a business problem and often leave security out because they view security as an IT function that puts up barriers to business, says Randall Gamby, an enterprise security architect for a Fortune 500 insurance and finance company.
"Security teams have insights into how data is protected and how access works," he says. "They need to be included."
Security technologies are often considered overhead infrastructure, but if left out of disaster recovery/business continuity planning, could mean users can't access the business resources they need in a recovery situation, he says. For instance, if the organization uses single sign-on in its routine business operations but SSO isn't supported in the disaster recovery plan, then users may not be given proper log-in prompts or be able to access certain backend applications.
Some companies, however, make security a priority in their disaster recovery planning. An information security officer at a financial institution, who requests anonymity, says his organization is in a highly regulated industry and cannot afford to overlook data security.
"Purely from the standpoint of being compliant with the regulatory bodies, it [security] has to be at the top of the list when we look at disaster recovery," he says.
Indeed, companies -- particularly those in highly regulated industries such as financial and health care -- need to be aware that data security mandates aren't waived in a disaster.
"We have tremendous compliance requirements from a variety of regulators," says the financial information security officer. "The requirements for information security don't make a distinction between whether you're in a disaster recovery mode or not."
In fact, the HIPAA Security Rule specifically calls out the need for maintaining security in an outage situation, Moyle notes. Section 164.308(a)(7)(ii)(C) requires the implementation, as needed, of procedures to enable continuation of processes for "protection of the security of electronic protected health information while operating in emergency mode."
One disaster scenario to consider is the possibility of guard staff reductions and loss of monitoring capability to prevent theft, Moyle says. If servers or laptops are stolen with regulated data on them, a company would still have to meet breach disclosure requirements.
"You could incur regulatory penalties over and above what it costs you from a downtime standpoint," he says.
Organizations don't tend to get audited during a recovery operation but they need to be prepared down the road, SunGard's Hughes says. "Now I'm getting an audit six months later. How do I reconstruct the chain of custody for the data and how it was protected in the time frame, if the auditor wants that?" he asks.
Temple-Inland's Engle says he can't imagine a company that has PCI Data Security Standard compliance requirements deciding to operate for two months without protecting cardholder data after an outage. "You will get driven out of business if you go for an extended amount of time without all the same protections you had originally," he says.
There are a variety of disaster recovery methods including hot sites, cold sites, managed service provider and cloud-based services. No matter the method, organizations need to ensure the security of the site they're failing over to, experts say.
"You're trying to replicate normal operations at a backup site… Make sure you have all the security in place when you get there," says Beau Woods, solutions architect for security and risk consulting services at Atlanta-based security services firm SecureWorks. That means making sure firewall protection, intrusion detection and antivirus are in place and updated, and if a company has a security operations center, making sure there's a place for those employees to sit, he says.
"You need to make sure that when people arrive to activate the site, that the controls in place are at least as strong as the controls that would be operating in a normal scenario," Moyle says. "The policy doesn't change in an emergency."
Gamby says companies often take it for granted that users have access to systems and forget about the access management layer -- such as virtual directory services, federated technologies, and containment zones -- that must be in place at the recovery site in order for business to continue.
"A lot of controls around data protection are based on a user's profile and that profile may get down to identifying the particular IP or MAC address for the system he or she uses," he says. "At a remote facility, you need to make sure those profiles are put in for those individuals so they can access the data from their desktops."
Organizations also need to consider encrypting the shared communication lines used for data transmission when switching over to a recovery site, Gamby says. After an incident, companies typically switch from their dedicated lines to a service provider's shared pipe to reroute traffic to the backup site. While the shared links won't mean cross contamination of data, someone managing the switching environment could look at the traffic crossing the lines, he says.
For BioWare, an electronic game developer, uptime and availability are critical -- as is data security, says Craig Miller, senior team leader of infrastructure. The company uses a virtual tape library for disaster recovery; the digitally replicated tapes are sent over an encrypted VPN tunnel to another site. Every couple months, physical backup tapes are encrypted and sent to Iron Mountain.
"Being in game development, all we have is our data…If the assets aren't available or recoverable, we don't have anything," Miller says.
BioWare uses two storage arrays from Compellent and plans next year to move one array offsite and double the disk size at each site for full cross replication; if one array goes down, the other could be active in seconds, he says.
If contracting with a fixed site disaster recovery provider, managed service provider, or cloud-based service, companies need to vet them as they would any third party, says Rachel Dines, an analyst at Forrester Research.
"You need to know where they are storing the data, what are their encryption, access control and authentication policies, and whether they can provide documentation for all that," she says.
Organizations usually will ask vendors if they use encryption but neglect to ask important questions about the type of encryption, where the keys are stored and who has access to the keys, Dines says, adding, "Vendors shouldn't have access to your encryption keys."
Third-party recovery sites raise the issue of multi-tenancy, which bring additional security concerns, Dines says. "I'm not sure if people think through all the full implications of that -- there are other companies' employees walking around there if they declare [an emergency] at the same time. You need to make sure the access controls to your infrastructure and data is strictly controlled."
SunGard's Hughes says customers in a shared recovery site need to step up their vigilance but acknowledged that can be a challenge. "That's tough in a recovery because that's not your first focus," he says. "The first is to get out of the situation you're in."
Cloud-based disaster recovery is relatively new but comes with a set of security concerns that organizations need to pay attention to, says George Ferguson, product marketing manager of security, compliance and continuity services at HP. The cloud-based option offers flexibility, cost savings and the ability to reduce recovery times, but companies need to step back and evaluate the cloud vendor's security controls, he says.
Ferguson cites the Cloud Security Alliance's guidance regarding the 13 critical areas of focus for cloud computing. Among the 13 areas is business continuity and disaster recovery, and the CSA recommends inspecting a cloud provider's recovery and continuity plans.
Missing backup tapes
A sample of breach reports involving backup tapes over the past two years
October 2010 -- San Diego Regional Center, which serves people with developmental disabilities, notified some clients that a backup tape created for the purpose of disaster recovery testing, was lost by UPS in shipping, according to a breach noticed obtained by PHIprivacy.net. The tape contained some current and former consumers' names, Social Security numbers, addresses and medical diagnostic information.
September 2010 -- Pediatric and Adult Allergy, P.C., in Iowa reported losing a backup tape with patient personal information in July. Information on the backup tape included names, Social Security numbers, and health plan data. The loss affected 19,222 individuals, according to the U.S. Department of Health and Human Services.
June 2010 -- Insurance broker Marsh and Mercer reported the loss of a backup tape that was being transported by a third-party courier, according to records obtained by DataBreaches.net. The tape contained employee benefits information; the data was maintained by Marsh's Assocation business, which operates through Seabury & Smith and Mercer Health & Benefits. The number of records exposed totaled 378,000, according to Privacy Rights Clearinghouse.
February & April 2008 -- Third-party couriers lose unencrypted backup storage tapes belonging to the Bank of New York Mellon in two separate incidents. The lost tapes potentially exposed the data of approximately 4.5 million people.
January 2008 -- GE Money, the firm hired by JC Penney to run its credit card operations, said it lost a backup tape containing the personal information of about 650,000 shoppers of JC Penney and other merchants. The tape was discovered missing in October 2007 by a worker at Iron Mountain.
BACKUP DATA TRANSMISSION
Disaster recovery has traditionally relied on tape-based backup to off-site storage, but the transfer of those tapes doesn't always go as smoothly as organizations expect. In recent years, there have been numerous reports of backup tapes missing in transit, resulting in breach disclosures. [see sidebar above]
Backup tapes are at risk in transit, but unlike BioWare, many companies still fail to secure them with encryption, experts say.
"We've come a long way in starting to secure devices like laptops, CDs and thumb drives, but when you look at the backup tape generated on a daily basis in a lot of organizations across the world…rarely is someone encrypting them," he says.
SunGard's Hughes says companies tend to focus on the process of maintaining backup tapes and having a third party transfer them rather than securing them. He's seen a shift away from tape backups, not necessarily for security reasons but because of concerns with recovery times. At the same time, the cost of replication is going down, he said.
HP's Ferguson says the security risks of lost or stolen backup tapes -- along with the need to improve recovery times -- has driven a move towards electronic vaulting services, also called cloud-based backup and replication, as a means of avoiding the physical transfer of tapes.
Overall, cloud computing has the potential to ease disaster recovery and business continuity by making it easier for organizations to have a mobile workforce, says Dean Ocampo, solutions strategy director at security supplier SafeNet.
"The benefit of moving to a cloud infrastructure is that you can access it from anywhere," he says. However, companies are reluctant to move their IT processes to the cloud until protections such as encryption and authentication are in place, he adds.
Designing a disaster recovery site has to be similar to anything else -- with security built in, says Temple-Inland's Engle.
For example, companies need to identify ahead of time potential areas where security controls constrain application functions or implementations and plan accordingly. If you know you had difficulties installing something in your primary environment then you should anticipate that it will be even more problematic in a recovery scenario. An organization doesn't want to find itself in a situation where it's trying to recover an application and has to shut down security controls to make it work, and then is unable to turn them back on, he says
"If you develop a disaster recovery plan and try to secure it on the backend, it's not going to work," he says.
The information security officer at the financial institution agrees that security must be integrated from the beginning.
"Our attitude is that we don't bolt on security -- it's baked in across the board, not just for day-to-day operations but for that disaster recovery situation, which is potentially a day-to-day operation," he says.
Marcia Savage is editor of Information Security. Send comments on this article to firstname.lastname@example.org.
Companies err in throwing disaster recovery planning onto IT and forgetting to test
Leaving security out is one of the mistakes organizations can make in disaster recovery/business continuity planning, but experts cite a couple other common mistakes: Leaving the planning to IT and not doing enough testing.
Companies often throw disaster recovery onto the IT team without prioritizing what business functions are the most critical to recover and setting recovery deadlines, says Beau Woods, solutions architect for security and risk consulting services at Atlanta-based security services firm SecureWorks.
"IT has to make decisions on its own and it ends up not being in line with the business," he says. "You need to have a cross-functional group make those high-level decisions before going down the road of how you'll recover from a disaster and continue business."
Another frequent mistake organizations make is not conducting enough test of their recovery plans, Woods says: "You need to make sure the way you've designed it is the way it operates in real life, both on the technology and people/process side."
William Hughes, director, consulting services BC/DR Center of Excellence at SunGard Availability Services, also says testing is critical.
"People tend to build a solution and think that's the end state, but that's really just the beginning," he says. "The end state is about four tests later, after you work through the bugs."