If you're an NFL fan in April, you're well familiar with mock drafts. These pretend exercises portend to make a...
best guess at whom your favorite football franchise will select on Draft Day. Granted, while teams may be worth hundreds of millions of dollars, the NFL isn't playing the same high-stakes game as the federal and state governments.
So when a state such as Delaware calls all hands on deck for a mock exercise simulating a coordinated attack on information systems and communications, there's more at stake than who will be taking snaps for the next 10 seasons. Lives, critical infrastructure and national security are on the line.
Delaware's Dept. of Technology and Information (DTI) had conducted tabletop incident response exercises since 2005 to great results. Year after year, new insight was gained into technology and processes that weren't up to speed or needed a tweak. But the tabletop format was losing steam and organizers feared what had long been an effective evaluation tool would lose its value. IT people in particular aren't engaged for long without the ability to bang on a keyboard, write scripts and see measurable results. That was incentive enough for the state last year to add a hands-on aspect to the drill.
"It's good to simulate attacks on the state's information resources so folks in various capacities of state government can play along and talk about response and what things we can put in place to perhaps prevent an attack from happening altogether," says the state's chief security officer Elayne Starkey. "It's good to practice; for the same reason you have fire drills."
Planning every step of the way
Delaware's exercise is anything but fire drill. To the contrary, it takes six months to plan the exercise, which involves 125 people from federal and state agencies, including IT managers, law enforcement, the FBI and academics. Disaster recovery coordinator Lisa Wragg is the project manager who drafts the exercise's objectives, organizes a steering committee that reviews and approves those objectives, and then, using the Homeland Security Exercise and Evaluation Program (HSEEP) as a model, plans out the sequence of events and milestones that must be met along the way.
There are four preliminary meetings under that model: a concepts and objectives meeting; an initial planning conference where the concepts and objectives are finalized and approved, the venue approved and participants selected; a midpoint planning conference where the sequence of events is established; and a final planning conference, where the review of the day's scenario and logistics is approved. The steering committee is a partner at each milestone, and that was made up of the state's high tech crimes unit, state police and the Delaware Emergency Management Agency.
"You have to create a scenario and put together outline of day's events. People need to have a reason why things are happening," Wragg says, adding that she used many of the lessons learned in DTI's three previous exercises to build this one. "If you just throw people in a room and just start hacking them and not have a story to go by or understand why something is happening, it's kind of meaningless to them."
Last October's scenario had a timely script. Held a week before the presidential election, the plot involved a cyberattack by the fictional country of Dystopia on states' voting machines. The plot was hammered out months earlier, and reinforced last summer when attacks on the country of Georgia's state-run websites were conducted prior to physical conflict during its war with Russia.
"That drove home the possibility of what could happen," Wragg says. "We needed to prepare for it. We needed the scenario to be a terror attack this time."
Simulated attacks, real responses
The simulated attacks escalated throughout the day; admins at first seeing only sporadic attacks against websites. But as the day went on, the attacks grew and it became clear that they were coordinated and politically motivated. The state's response plan kicked in and ultimately, a simulated state of emergency declaration was made by the governor.
Starkey says the attack scenarios are kept close to the vest with fewer than 10 people knowing what's about to take place. The added dimension of this exercise being a terrorist attack on the voting infrastructure required some careful treading. Starkey did not want to leave the impression on any of the participants -- including the National Guard, Air Force, school districts, state police, FBI, Dept. of Transportation, Dept. of Labor, in addition to DTI--that the state's election system was vulnerable.
All of the players were present at the DTI emergency operations center on Oct. 29 for the exercise, and in her opening remarks, Starkey laid out the day's high-level goals: prevent cyberattacks, sharpen response procedures and recovery.
Strategy: Three keys to success
Understand the threat landscape and plan your tabletop exercises accordingly.
Motivated attackers are going to penetrate even the most ardent defenses. Companies that realize that this is the information security environment of 2009, are the ones realizing the need to run through functional and tabletop incident response exercises such as the one conducted by the Delaware DTI.
Lenny Zeltser, an incident handler with the SANS Internet Storm Center, says even enterprises with mature security practices find great value in these mock exercises. He defines three keys to success:
- Define your success criteria. "You need to define what it means to do well," Zeltser says. Have you responded to an incident within 30 minutes, and have a good sense for the scope of an attack either hours later? Or maybe you define success as learning within a pre-determined period of time what data was affected and whether the right people were notified and put in position to make decisions.
- Involve the right people. "It's too easy to operate in a silo," Zeltser says. You might be one of 10 teams responding to an incident, and those nine other teams won't prioritize security the way you do. "That means you may have to have power or authority or good will to get them involved."
- Evole your exercise. "Don't run through the same exercise every year," Zeltser says. Your incident response exercise should evolve just as your business changes, the economy grows or shrinks and security priorities change.
"One thing that was important to us, was that when we start the exercises, that we create an environment of trust, take away the threatening feeling in the room--dispel that right away," Starkey says. "In my opening comments, I stressed this was not real. I wanted them to feel like this is safe haven, and that we understood they were all at different points of readiness."
"Don't feel badly about not having a policy in place that you should, or a procedure not defined completely. This is the place to kick all that around," Starkey adds. "One of my key objectives is for them to leave that day with a little to-do list of things they want to take care of in the weeks after the exercise. We want them to each year to go away with ideas of things to do to strengthen their infrastructures, and to improve their ability to respond and recover from an attack like this."
At an appointed time, programmers and network security engineers began releasing attack scripts against websites that were built in a development environment set up on a segmented network. Responders in the EOC would need to recognize problems with a site such as defacements or denial-of-service attacks and take appropriate countermeasures, which were evaluated.
"It was like a little NASA -- rows and rows of computers, screens up on a big wall where the participants were sitting, and behind the glass was exercise control where the injects and scripts were released," Wragg syas.
Website defacements were the first wave of attacks, launched against the home pages of various state agencies. As word spread of the attacks, other agencies began to take measures to harden their Web apps to avoid being taken down as well. Several, Starkey and Wragg said, beat attackers to the punch.
"That was incredibly motivating to the other agencies," Starkey says. "We highlighted it in one of the breaks and congratulated them on the good work they did."
In another room adjacent to the EOC, a tabletop style scenario was set up where people of similar function would work together. The service desk was also there taking incoming calls for trouble tickets. As soon as the attacks happened, calls flooded the service desk. High Tech Crimes officials were at one station, and working with law enforcement, they quickly began tracing the source of the attacks. Meanwhile, the state's Joint Information Center (JIC) featuring public information officers from different state agencies were at another putting out coordinated media releases and crafting appropriate public responses, alerting citizens that they should take caution using agency websites.
"It was pretty cool and interactive," Wragg says.
Once that segment of the exercise was complete, the DTI held a quick briefing on the importance of preserving evidence. Admins are initially more concerned with the availability of systems and getting them back online, but in this instance, they had to tread lightly to preserve the integrity of the scene and assist in tracking the source of the attacks. The participants were also evaluated on how well they used the state's incident command system, prescribed by the federal government. The framework is built for emergency management agencies and represents a set of standard response procedures.
The next wave of the attack involved more website attacks, this time the target was sensitive personal data. Simulated FBI warnings were sent out that terrorists had launched cyberattacks against critical infrastructure, and soon thereafter, calls began flooding the service desk with citizens reporting possible identity theft after accessing services on state agency websites. The response involved assessing the cause of the breaches and reviewing data protection procedures. JIC also worked up statements directing citizens how to protect themselves online, and if necessary, report incidents to police.
The final phase of the exercise combined another hack with a physical attack. Denial-of-service attacks were launched against agencies' sites and services, while simultaneously terrorists were disabling lines used by service providers statewide. The offshoot was that these attacks could possibly impair the state's ability to vote in the upcoming elections. Steps were taken to rapidly move critical infrastructure to redundant facilities and keep services available until the service providers to could complete repairs.
"The exercise creates a lot of interest in updating plans and going back and checking websites, making sure they're up to date and patched," Wragg says. "There is a lot of after-exercise activity. People want to do something."
Measurable metrics and reviews
Being the fourth such exercise, many incident response processes are mature. Media and external communication are solid, Starkey and Wragg note, while adding that internal communication between agencies is an ongoing process.
"If we're looking for measurable stuff, some agencies quite frankly need help, and we're going to help them," Starkey says. "Quite frankly, I don't think we would have been able to identify who needed more help than others until we did the exercise."
Starkey says the agencies did well against the four stated objectives. All agencies identified vulnerabilities in their infrastructure leaving them susceptible to Web-based attacks. Each agency had a prescribed process for defending against attacks and rolled out those processes accordingly. Each addressed the preservation of evidence, with different levels of maturity in their respective processes. This was an area Starkey says ongoing education will be key going forward.
Tabletop Exercise: Lessons Learned
Lisa Wragg, disaster recovery coordinator for the Delaware DTI, was the project manager for last year's incident response exercise. She lays out seven lessons learned.
- Assign a project planner
- Secure an executive sponsor; CSO Elayne Starkey was her sponsor
- Follow a master event list and build your scenario around that list
- Stick to your scenario; what look like minor changes could have bigger impacts down the line.
- Outline the details of your scenario, including attack scripts
- Address current threats in your scenario.
- Get an outside agency to assess how you do; SunGard's Incident Management Exercise Service did DTI's assessment
Business continuity is also another area DTI will concentrate on going forward. The coordinated physical and cyberattack that played out in the final phase of the exercise stressed the importance of a continuity plan for critical services such as voting that must continue seamlessly should a key state network fail.
Breach notification was the final goal that each agency met with flying colors, much to Starkey's satisfaction since each agency information security officer was given a procedure to follow on notification. Service desks were overwhelmed with calls; an indication the procedure was being followed.
In the end, Starkey says adding the functional component was definitely a touchdown, and that last year's participants would never go back to just a tabletop exercise.
"We have a catchphrase about this being a journey to compliance," Starkey says. "I recognize we're not there, we're not at 100 percent compliance across the board. We do see everyone moving different rates."
"If you look at the write-up after first year's exercise, the objectives were fundamental about increasing customers' awareness that cybersecurity was important. We've made incredible strides there to get them to pay attention, let alone comply with a 41-page security policy."
About the author:
Michael S. Mimoso is Editor of Information Security. Send comments on this article to firstname.lastname@example.org.
Dig Deeper on Information Security Incident Response-Information
Finnish government supports local authorities in cyber security initiative
Cooperation and exercises key to cyber defence, says Nato centre
SQL innovator Jim Starkey journeys into the database future
Q&A: Database infrastructure must match modern apps