Brad Arkin was so stressed in the weeks after the high-profile Adobe data breach that he grinded his teeth to the...
point where he cracked two of his molars during incident response meetings.
"Not a fun situation," he said.
Arkin, vice president and chief security officer at Adobe Systems Inc., recounted his experiences of the data breach aftermath during a presentation at the Privacy. Security. Risk. 2015 event in Las Vegas last month. Arkin had only been on the CSO job for a few months before the company's data breach; he had previously served as senior director of product security and privacy at Adobe and was promoted to the CSO role in April 2013.
"Basically, I'm on the hook for everything," Arkin said. "That's very satisfying for our customers because there's one throat to choke."
Seven months after Arkin's promotion, a nightmare scenario was realized and customers' hands were soon fastening themselves to Arkin's neck. On October 3, Adobe announced it had discovered an unauthorized intrusion into a database server and other machines in Adobe enterprise network. The net result of that intrusion, he said, was that certain data was copied and removed from the server, including encrypted credit card numbers, customer passwords, unencrypted usernames and email addresses, and some portions of source code for Adobe products.
Brad ArkinVP and CSO, Adobe
"There was a lot of media coverage of [the incident], and that media coverage went on for quite a while," he said. "A security incident is not like a car accident where it happens and that's it. The reality is the incident itself unfolds in a very slow manner."
The forensics investigation work took weeks, Arkin said, but that was only one part of the post-breach effort. A seemingly endless string of meetings were held about addressing the media, notifying customers, handling potential legal issues and revamping Adobe's infrastructure and security program. Some meetings were every four hours for forensics updates, Arkin said, while other meetings were daily affairs to update the team on the latest information.
"There are a lot of different moving pieces with different teams in the company who have different concerns" Arkin said, citing engineering, legal, media, communications and investigations. "And sometimes those concerns are competing for priority, and sometimes they are very well synched in how these teams can work together."
One of the biggest concerns was figuring out what product source code was stolen, and what the repercussions of that theft could be.
Adobe data breach: Impact of source code theft
Since the Adobe data breach exposed product source code, the product engineering team had to work closely with the investigative and forensics teams (which include both internal Adobe staff and third-parties).
"There are engineering teams that had work to do as a result of what the forensics team was revealing," he said. "So as information was coming out, the engineering teams might have to make changes to the [product] code or operational environment."
The source code theft proved to be one of the more difficult aspects of the Adobe data breach. Arkin and the investigative teams quickly tried to determine what effect the source code theft would have on Adobe customers, if any. That involved a deep analysis of the Adobe products as well as examining third-party software programs that had their source code exposed, either intentionally or inadvertently, all of which was very time-consuming.
"We sat down and really thought through what was taken and what we thought the impact was and what bad guys do in the real world," Arkin said. "And we came to the conclusion that we weren't aware of any specific increased risk to our customers as a result of that source code theft."
But that didn't stop many from speculating otherwise, Arkin said. "There were plenty of experts that the media was able to find and quote who offered different opinions," he said.
Brad ArkinVP and CSO, Adobe
With the benefit of two years' hindsight, he said, the so-called experts were proven wrong and Adobe's stance on the source code theft was proven correct.
"We never saw any zero day attacks or increased attacks against the products that had portions of the source code taken," he said.
Arkin said Adobe had to balance the need to arm customers with the best available information with the potential risk and liability of making public statements that weren't firmly grounded in the facts. He also said it was a challenge to determine when to make public statements or customer notifications.
"You don't want to wait six months until every fact is out because then the actual information isn't as timely for the people who need it," Arkin said. "But if you go out too soon, you confuse people."
Adobe, for example, initially estimated that approximately 3 million customer accounts were affected by the breach. But several weeks after its initial announcement the company announced the breach likely affected 38 million accounts.
Adobe data breach: Too many moving parts?
Another thing Arkin learned was the risk associated with having so many teams moving at once with a lack of clarity about who's ultimately in charge and who are the primary decision-makers, which he described as the "too many cooks in the kitchen" phenomenon.
Adobe's internal investigation effort was led by the company's Security Coordination Center, which Arkin called the "uber incident response team" that handles different aspects of the Adobe security posture. But the investigation effort included many third party vendors and consultants, and that created "the potential for a lot of friction," he said.
For example, Arkin said, third party organizations might overstate claims and make statements before they have the full set of facts, while internal teams may be more reluctant to definitive statements even after determining what happened. CSOs, therefore, need to manage the different groups in order to get all available information and assess it in its content.
Ultimately, Adobe and its third party partners completed the forensics investigation and successfully determined the affected areas of the network, the specific data that was exposed and copied, and that the attack was conducted by external actors and not insiders. "The forensics investigation brought us quite a bit of clarity about the incident," Arkin said.
To successfully manage the many different teams and moving parts during a data breach response, Arkin recommended that enterprises conduct table-top exercises for decision-making models so that when a security incident occurs, they're not dealing with a lack of clarity about who's in charge. He also said the table-top exercises on data breaches were valuable for executives and managers outside of IT and InfoSec who don't have a lot of experience or knowledge around security incidents.
"If you've seen computer security on TV, you get the answer right away and there's no confusion," he said. "In the real world, you'll get an answer from forensics that appears definitive. And then four hours later, they say "No, we were wrong, sorry. This is now the right answer." And for very high level folks who are used to getting real clarity, that can be very jarring."
Adobe data breach: Notifying customers, handling the media
Even though customer credit card numbers and passwords were encrypted, Arkin said Adobe decided making a public statement about the breach and informing customers was the best approach.
"From a business perspective, we decided the right thing to do was to notify them," he said. "We figured our customers would want to know that there was a breach and that their credit card numbers and passwords were encrypted."
But Arkin said there is often a tug of war when it comes to data breach notification between legal teams, public relations, incident response and customer management teams.
"There's this tension at times where the theoretical risk minimization from a legal perspective is sometimes at odds with what the expectation is from our customers," he said. "So we didn't want to hide up in a corner and say, 'We're not going to do anything, we don't have to,' because that's not going to play well with customers."
There was a lot of work to be done on the mechanics of how to notify customers, and determining legal risk and liability was only one aspect. The incident response team had to figure out not only the right information to disclose to customers, but also figure out the best way to notify millions of people that were spread across different states, countries and regions. The team also had to figure out when to ultimately go public, when to notify customers of the breach, when to have the customer care information live on the Adobe website and how to handle inevitable questions from the media. "The logistics of this were quite significant," he said.
Arkin said Adobe tried to make public every potentially actionable fact it had in order to present a clear picture of what had happened and what the potential risks were for customers, but the company often found itself at odds with the media. "We ended up in a situation where we were chasing after media reports trying to correct inaccuracies, but each correction creates a new day of coverage," he said.
Brad ArkinVP and CSO, Adobe
The net result was a sort of "fog of war" around the Adobe data breach, he said, that ended up confusing customers. This led to the public relations and communications teams constantly updating FAQs, correcting inaccurate reports and working with media members to keep the confusion to a minimum. But Arkin said Adobe made the conscious decision not to tackle reports or quote third parties that had the facts correct but were coming to conclusions that Adobe found to be far-fetched.
Nevertheless, Arkin was critical of how security vendors often offer up experts to provide insight into high-profile hacks, data breaches and security incidents.
"The person that sells network monitoring solutions is saying, 'Well, you should have had more networking monitoring, you would have caught this sooner,' and the guy that sells code security solutions is saying, 'These guys should be using my code scanner' and so on," he said, which for the average customer only contributes to more confusion.
Arkin's advice for enterprises included having a centralized public relations team to handle all information and media requests related to the incident, having a dedicated spokesperson who is properly trained and experience with speaking to media members in such situations, and having a PR-specific incident response plan in case of a major breach.
Adobe data breach: Managing staff, stress
One of the more overlooked aspects of dealing with enterprise data breaches, Arkin said, is managing stress levels of both your staff as well as yourself. Adobe employees were working 18 hours a day, seven days a week for three straight months, and the situation put massive pressure on Arkin and his staff.
"Expect the stress. Acknowledge it," he said. "Find ways to manage it, and don't just bottle it all up."
Arkin, who described himself as a "teeth-clencher," didn't do that, and it led to two cracked teeth in the fall of 2013. In addition, the stress took a toll on his physical appearance; during his presentation, Arkin jokingly referenced his appearance with "before and after" slides of Brad Pitt -- one photo showed a handsome and perfectly groomed Pitt, and the other showed a bearded, disheveled-looking Pitt.
Arkin advised the audience to take the proper steps to relieve stress before people run out of gas and start making costly mistakes. Adobe's security team didn't start implementing stress relief steps until it was well into the data breach aftermath. Some of the steps included rotating staff members and ordering overworked people to take some much-needed time off and get away from their laptops and phones as well as the office. If a staff member can't be rotated out, Arkin said, then the company needs to provide additional support to that person.
"If an engineer from Seattle has been living in a hotel room in San Jose for two months, maybe he misses his family, so maybe we can fly him home to Seattle for the weekend or bring the family down," he said.
Another important step was reducing the responsibilities of people who were dealing with the data breach by passing off their normal duties and tasks and delegating them to other employees. Arkin also said people should do their best to stay in their normal daily routines of exercising, eating healthy and getting enough sleep.
Adobe's human resources department worked with executives and incident response team leaders who were dealing with stress during that period, which involved a lot of "armchair psychologist stuff," Arkin said. "The day they told me I was going to be the one to sign my name on 3.5 million [breach notification] letters, HR pulled me aside and gave me a chance to talk about my feelings," he said.
But HR wasn't available for lower level overworked and stressed employees during that time. Therefore, that responsibility landed on Arkin and his administrative assistant, who adopted a "caretaker role" to monitor staff members and point out when someone looked like they needed a break.
"I'm a little bit of robot," Arkin joked. "So it was her job to notice if people around me were falling apart."
Adobe data breach: Cleanup and reboot
After notifying customers, grappling with the media, and performing a full forensic investigation, Adobe had to wipe the slate clean and start over.
"In the short term, you have a bunch of machines that were inappropriately accessed and accounts that may have been compromised, and so you need to take all of that and basically burn it to the ground," Arkin said. "There's a science to how you can actually eradicate an adversary from your environment, and if you do it wrong, they may be able to burrow away and stay persistent [inside the environment]."
That involved wiping client devices and rebuilding servers from scratch and not just resetting passwords but destroying accounts completely and creating entirely new ones in their place. But Arkin said enterprises shouldn't rebuild their IT environment in the exact condition it was in before because the process gives security managers an opportunity to improve controls, policies and practices to prevent similar breaches in the future.
For example, Arkin said, the database that was accessed by the attackers held encrypted credit card numbers of customers. During the rebuilding process, the security team decided to review that practice. "We don't have any evidence that the bad guys ever decrypted those," he said. "But still, having encrypted credit card numbers around was something that made us uncomfortable, so we moved to a tokenized system."
Arkin said there are no longer any encrypted credit card numbers in the system that was breached. Instead, the company uses tokens to verify and complete transactions with payment providers. That change required a lengthy project, but one that was ultimately worth the effort, he said.
"Tokens have no value or use to bad guys because all they could do is basically run transactions, which would then send the money to us," Arkin said. "So there's no real incentive for the bad guys to attack those particular slices of data."
In addition, Adobe also consolidated the company's many different security teams into a more cohesive group under Arkin, who said he was able to implement a clear information security vision across the company and control budgeting for security needs (Arkin said he currently spends about 80% of his budget on the prevention side and 20% on the reaction side). In addition, he said having a cohesive group with a combined budget allows him to assess potential areas of need and shift resources as needed to address them so that there are no glaring holes in Adobe's security posture. "That allows us to make apples-to-apples risk trade-offs across the company," he said.
Overall, Arkin encouraged audience members to think of security and data breach response in a bigger picture that includes not just technical aspects but legal, public relations, human resources and management aspects as well.
"Having a wonderful defensive posture and zero effort put into your PR strategy if there's a breach is an unbalanced application of resources," he said. "You need to think about both sides."
Find out why experts expressed concern about cloud data security at the Privacy. Security. Risk. 2015 conference.
Learn about data breach response horror stories from the frontlines.