tharun15 - Fotolia
It's the middle of the holiday season and your servers and network are running at full capacity when you notice a domain controller suddenly acting weirdly. Soon, users start complaining about errors in accessing resources. Someone wants to know why users are getting locked out of their accounts. All sorts of suspicious login activity are going on across the network. Before you know it, there's a full-blown crisis on your hands that your security team cannot manage on its own.
If you are like many other organizations, this is probably when you might bring in an emergency incident response provider to help handle the crisis. Professional IR teams basically provide incident handling services for fees ranging from a few hundred bucks per hour to tens of thousands of dollars, depending on the scope of the breach and the amount of work that needs to be done to remediate it during the incident response process.
Providers of professional IR services can quickly bring the additional resources and the expertise that companies often need to handle a rapidly unfolding threat. But there's a lot you need to do to get the best out of these services, and that begins with a clear understanding of how the emergency incident response process works and what to expect when you hire an IR provider to handle an ongoing crisis.
Scoping the problem
The first thing an IR provider is going to want to know when you call them is as detailed an explanation as possible of what's going on. Organizations that are in the midst of a multilayer malware attack or network intrusion often do not have a full idea of the origin or scope of the problem. Still, it's vital to gather as much detail as possible. "Normally, when a customer reaches out to an incident response company the first thing they are going to want to know is what is going on," says Bob Shaker, director of strategic operations, cyber readiness and response at Symantec.
An IR provider will want information on why you think your organization has been compromised, when you discovered it, how it was discovered and whether it was first spotted by someone internally or reported to the organization by the FBI, law enforcement or some other entity, like a credit card company.
It's vital in the scoping phase to have individuals from the organization who know exactly what kind of information they can provide the IR provider in the form of log data and other forensic evidence, says Jim Aldridge, security-consulting director at FireEye's Mandiant incident response unit.
The IR provider will use the information the client provides to get an idea of the scope of the security incident and to decide what kind of resources, including onsite staff, might be required to address the issue. "When an organization contacts an incident response provider they should have a discussion that conveys the scope of the problem, as they know it, to make sure that the consultant understands the situation they are getting into," Aldridge says. "This is especially important so you don't get mismatched expectations."
The contracting phase
Once the IR provider has had an opportunity to assess the situation, their next step will be to provide you with some kind of a quote or estimate for what they think it will take to handle the incident. The proposed contract should typically contain a detailed explanation of the services they will provide, including whether the IR provider will actually help remediate the problems or help you identify them so you can fix them yourself.
It's important in this hiring phase to work out and understand the documentation, access and knowledge that the service provider will need in order to handle the incident, according to Christopher Pierson, chief security officer and general consul at Viewpost, an online payment platform. "This is critical to ensuring the right resources are brought to bear," he says. Companies that have apps and services running in the cloud are sometimes restricted in the kind of forensic parties they can bring in to investigate a incident, so it is important to work through such details ahead of signing the contract.
Also, it's vital to find out are what kind of investigative skills the IR provider has as well as the technology, tools and threat intelligence it can bring to bear in handling a security incident, adds Sanjeev Sah, director of security and chief security officer at Texas Children's Hospital.
Generally, it is a great idea to engage an incident provider on a retainer basis before you actually experience a breach. (See: Four Tips for Getting the Out of Your IR Provider.) Then you don't have to waste critical time in working through such details in the contract process or in explaining your organization's incident response process in the middle of a major crisis.
The last thing you want is to have to find and chase down someone in your organization with the authority to sign off on the contract when an incident is unfolding, says Symantec's Shaker. "When there is a crisis the person who should sign off on a contract is usually in a war room somewhere and now you have to chase them down."
Make sure to keep your insurance company, general consul and other stakeholders informed on what's going on so everybody is on the same page, he says.
Investigating the issue
The IR provider will need whatever information you can provide, like system and network logs, network layout diagrams, system images, network traffic behavior and more, to piece together what might be going on.
Often there's a tendency by organizations to panic when things start going wrong and the gut reaction is to shut systems down to prevent more bad things from happening to them. Awful idea, says Shaker. "The number one important thing is not to power things off. Once you shut things off you have actually erased a considerable amount of evidence, especially the memory resident stuff."
The investigating team uses the information your organization provides, as well as information it gathers on its own from endpoints and other sources via proprietary interrogation tools, to identify and document file names, file hashes and other threat indicators, says Kevin Strickland, senior incident response consultant at Dell SecureWorks.
This is the part where the IR provider usually is able to inform the organization what happened, how the intrusion might have started or how malware was introduced on the network, what tools the threat actors are using and what needs to be done to contain the issue. "We are going to provide this information and tell them here is the action we need to take," says Strickland. If the recommended options are difficult, there can be some back and forth at this stage, he says.
Four tips for getting the most out of your IR provider
Engaging a third-party emergency incident response provider can help organizations quickly contain a developing security incident but they need to be ready to take advantage of the specialized skills such services bring to the table. Here are four tips on what you need to do to get the most out of your IR provider.
Have a plan
It's important to have a security incident response plan, exercise it regularly, and have all your partners selected before you actually need any of it, says Christopher Pierson, CSO and general counsel at Viewpost. The incident response process should already have the internal team in place populated with representatives from privacy, legal, security, PR, technology and the executive management function.
Know what to ask
Make sure you know what questions to ask before selecting an IR provider, says Sanjeev Sah, director of security and CSO at Texas Children's Hospital. Before signing up with FireEye's Mandiant incident response service, Sah verified their track record, and made sure the service provider had the technology and the threat intelligence capabilities needed to handle major incidents in the healthcare industry.
The organization's bench strength is an especially important consideration, according to Sah. When you have an incident, it is vital that the IR responder is able to put a dedicated team in place if necessary to help contain the issue. The third-party provider should be able to treat your incident as the most important one they need to handle, he says.
Don't wait for an incident to start looking for a third-party IR provider. Instead, hire an IR provider and place them on retainer for when needed. Get to know their leadership teams and make sure they are familiar with your security incident response plans and processes.
Consider having at least two potential partners on standby, Pierson adds. "It is great to work the high level plan out ahead of time, to work with the selected vendor ahead of time and share the security incident response plan with them, and even better - to practice a table top exercise with this provider and the teams on the ground so that when something happens the response is well known."
Make sure you have the information your IR provider needs in order to respond to a developing situation. You need to be able to readily make available system and network logs, network diagrams and topologies, incident and event management data, network flows, user activity logs and inventory management information.
"If a company does not have accurate and up to date network diagrams or data maps, the job of the incident response party is that much harder," Pierson says. "It is also advantageous to review the incident response plan with an external team, internal partners, and others so that the right tools are present in the environment ahead of time to better facilitate this task."
Containment and remediation
The team responsible for remediation and containment often works in tandem with the team doing the incident investigation, according to FireEye's Aldridge. "We have two workstreams: One is an investigative workstream aimed at addressing what systems, what data, what accounts may be compromised; the second is remediation." As the investigating team learns more facts about the security incident, it feeds the information to the remediation workstream and together with the client works on addressing the identified issues.
It's vital during the containment and remediation phase not to tip your hat off to the attackers. When attackers know they have been spotted, they often tend to take evasive action that may end up driving them deeper into your network and making them even harder to find in the process. "You want to make sure the attackers don't know you are on to them,' says Strickland. "It is very important to understand what is happening before you make any drastic changes."
The reports that the IR provider delivers once the issue has been properly contained, together with any recommendations, are vital to conveying the nature and scope of the breach to all stakeholders.
The incident responder needs to be able to communicate what happened and what they did to contain the situation during the incident response process in a clear, concise and jargon-free manner. Accurate language is vital to ensuring that a situation is not mistakenly downplayed or overhyped. A security compromise for instance is different from an actual network breach, and calling one the other can detract from the quality and accuracy of the report.
Finally, the third-party provider should treat your incident as the most important one they need to handle, says Shah. "It helps make sure that a team is available when you need them, on pre-negotiated terms."
Jaikumar Vijayan is a freelance writer with over 20 years of experience covering the information technology industry. He is a frequent contributor to Christian Science Monitor Passcode, eWEEK, Dark Reading and several other publications.
What is the CISO's role in incident response?
Should your organization respond to an extortion scheme?
How to avoid common incident response blunders