Recently, construction work near one of our offices resulted in a major fiber cut, taking out communications for the entire office. The call center at that office anticipated a huge call volume that week, and operations staff had a full plate of jobs to run. Thankfully, the crisis team was quickly gathered, command and control was established, response teams were called in to action, backup teams were engaged at alternate locations, and crisis was averted. Strangely enough, just a month earlier, another comparable crisis had hit the firm. Are we especially accident-prone? No -- we just act that way, once a month, via a quick crisis tabletop exercise.
We've found a good tabletop exercise is an opportunity to test our "best laid schemes" -- system recovery plans, business continuity plans, incident response plans, etc. -- with real-world scenarios thrown in. We take a real risk, and then consider what might happen when other things go wrong at the same time, or when critical team members aren't available. We add other teams to the mix, and the teams learn from each other about risks they hadn't considered, or expertise they might share.
Planning and executing exercises has been a great way to extend my role beyond traditional information security or risk management. I have asked team leaders, executives, board members, trusted
Sometimes, emergency tabletop exercises serve to provide reassurance or education. A board member may be particularly concerned about an incident that occurred at a competitor. Running an exercise may help to demonstrate the processes and teams that are in place to handle just such an incident.
About Matthew Todd
TITLE: CSO and Vice President, Risk and Technical Operations
COMPANY: Financial Engines
- Manages the privacy, security, availability and performance of production systems servicing over 7 million individuals
- Responsible for implementation and maintenance of systems and networks, including diverse secure data connections with eight of the largest U.S. financial services providers.
- President and board member, San Francisco Bay Area InfraGard
- Worked with U.S. Department of Homeland Security on the NETGuard pilot program, an effort to help local communities respond and recover from attacks on IT systems and communications networks.
- Participated in initiatives to engage SMBs in public-private partnerships such as InfraGard
An exercise is an opportunity to talk to peers about what bothers them, or even about scenarios they've seen tested. One peer offered this chilling but realistic scenario: A bomb threat is called in to building XYZ. After playing out the response plan, which involved immediately evacuating the building to the parking lot, then came the next phase: A sniper on a building across from the main exit. There are parallels in the IT space, such as disgruntled IT staff with access to sensitive data and the skills to set a logic bomb. Sometimes a good exercise requires a suitably devious mind.
I have had the most successful exercises by keeping a few key ingredients in the mix:
Objective – Every exercise needs a clear and realistic objective, consistent with the time and teams involved. A 20-minute tabletop exercise may be fine to test the crisis management team's initial response to a media scandal or a market crisis, but would not be enough time to test the multi-team and multi-day response that might ultimately be required. If we make the objectives clear up front to all concerned, it allows the players and the coordinator(s) to identify when a topic is best set aside for another time to keep the exercise on track.
Participants – A good exercise will include one or more coordinators who present the scenario, a scribe to record the events, and the right players from across the enterprise. The coordinator(s) are prepared and have the respect of the players. The players are ready to be fully engaged in the exercise, and include individuals or managers who would be involved in the incident, even if they become "victims" along the way. Even a victim can offer observations during the exercise. When the building collapses and takes out the CFO, she may be the only one who remembers that key public filing that she needed to sign.
Realistic scene – It's critical to devise a scenario that could really happen. Alien invasions seem somewhat implausible, but just as implausible may be a scenario that exactly follows the procedure the IT team has outlined for the failure of the central email server. A more realistic scenario would be the failure of the email server at exactly the same time the entire IT staff is focused on a major virus outbreak. It may seem unfair to throw a curve ball, but reality is like that. This is the perfect time to test for resource constraints: "You say John Smith would do this next step. What if John is on vacation? Ah, Jane would take over? Well, she just came down with the flu." Consider that both virtual and real virus outbreaks can occur over the holidays.
Hotwash – Each exercise should conclude with a "hotwash," where the players and coordinator(s) gather to discuss what worked, what didn't, and what needs to happen next. Were the objectives met? What did the players learn? Are there gaps in process, procedure, people, systems or communications?
So how do I know my exercise program is successful? There are a number of strong indicators. I have had the opportunity to engage with key partners in exercise plans, and help them consider risks in a more realistic fashion. The board has taken a keen interest in the program and its results. The CEO has incorporated a mini-tabletop exercise as a regular event for his senior staff. But ultimately, exercises have resulted in meaningful dialogue and action, and participants have gained confidence and skill in how to respond to real-world adversity.
The Security 7 Awards recognize the efforts, achievements and contributions of practitioners in the financial services/banking, telecommunications, manufacturing, retail, government/public sector/non-profit, education and health care/pharmaceutical industries. Click here to learn more about the Security 7 Awards and to see a list of all the winners.
This was first published in September 2011