This story has been updated.
The U.S. Office of Personnel Management handles all the health care and retirement data for millions of past and present federal workers. So while it may have an annual IT budget of about $200 million -- small compared to the Defense Department’s estimated $35 billion -- the agency manages very large databases with sensitive information about the personal lives of federal employees.
“We also handle all the background information and resumes for people applying for federal jobs,” says Jeff Wagner, director of IT security operations at OPM.
If security teams were only fighting teenage hackers looking to break into systems for the challenge or even the for-profit attackers, many of whom give up if it costs them too much to hack into a system, most organizations could handle the load. What troubles Wagner and other security professionals are the state-sponsored attacks by the North Koreans, Chinese, Iranians and Russians. He says the Sony hack last year in which the North Koreans were implicated really changed the game: “They throw unlimited resources into cracking systems with seemingly unlimited time.”
On June 4, OPM disclosed a data breach that may have exposed the personal information of 4 million current and former federal employees. The compromise has been linked to state-sponsored Chinese hackers, according to U.S. officials. Earlier intrusions, including a 2014 compromise that was also traced to Chinese hackers, caused the agency to update its cybersecurity tools, which helped them to uncover the current breach.
With only eight engineers on staff, Wagner looks for every way possible to automate endpoint detection and remediation tasks. It can take several hours by the time a security analyst gets an alert, sends it to a sandbox, pulls the threat intelligence feeds to learn what the malware does, creates a ticket to bring the user a new machine and then reimages the computer. And this is only for one event. “Most large organizations are hit with malware dozens of times daily, so it’s literally impossible for the typical stretched-thin staff to remediate each piece of malware,” he says.
Automatic response essential
Frank Dickson, a research director at Frost & Sullivan who covers the IT security market, says security professionals now realize that antivirus software alone won’t protect their networks. “What organizations need are tools that can cut a remediation down to minutes instead of spending two to four hours,” he says. “Having a tool … that can automate the remediation process is huge because it can take one IT person a couple of hours to do that by hand. That’s downtime with the user waiting and scarce security professional time that companies can’t afford.”
Fortunately, there’s no shortage of tools security professionals can use to protect their networks and endpoints, and remediate attacks. Some of the tools have been on the market for several years, such as Cisco Sourcefire, FireEye (Mandiant), Palo Alto Networks’ WildFire and Traps, and Digital Guardian (formerly Verdasys), which offers an endpoint detection and response module as part of its flagship data loss prevention platform. And there are other relative newcomers in the market, including CSG Invotas, Resolution1 Security (formerly part of AccessData Group) and Bit9 + Carbon Black.
What’s new in the endpoint threat detection is a move away from signature-based tools toward better integration with legacy systems to correlate network, endpoint and log data -- that is, an effort to streamline continuous threat analysis and automate incident response.
Wagner says his IT team uses a mix of tools, but executing the back end of a remediation with incident response software from CSG Invotas has helped him tie together all the products. Invotas Security Orchestrator lets them set up workflow templates that automate the remediation and reimaging process. “The templates are sort of like a Microsoft Vizio process,” he says. “Once I set up the workflows, all of this can happen inside of 10 seconds.”
Now the company runs HP ArcSight logs to search for malicious data. When the ArcSight security event and information management environment finds something, the IT security team will do packet capture based on custom programming, which will then trigger a FireEye Endpoint (HX series) alert once it has identified the malicious code. FireEye will put the code in a sandbox, and it will be compared against all the threat information.
At that point, an OPM security analyst will make the decision to start the remediation process in CSG Invotas Security Orchestrator. Assuming the security technician pushes the green light, the software disables the VPN, inactivates the user’s machine and creates a work ticket in BMC Remedy IT Service Management Suite to delete the malware and reimage the user’s computer.
“While it’s possible to automate the entire process, for the time being we have decided to still have a human being decide when to enable the remediation process,” Wagner says.
Prevention, detection and response
Jon Oltsik says he grows weary of hearing about the failure of antivirus software or the renewed interest in endpoints and remediation.
“Companies need three things today,” says the senior principal analyst at the Enterprise Strategy Group. “They need products that can deliver prevention, threat detection and response. Companies can’t do endpoint security in isolation; it has to be an integrated response.”
Golan Ben-Oni, CSO at IDT Corp., a telecommunications, banking and energy conglomerate in Newark, N.J., also uses a mix of network, endpoint and remediation tools to protect the global organization’s network.
Focus on the Endpoint Visibility
Remediation experts Fidelis Cybersecurity are well known for their network analysis tool Fidelis XPS, which they use on high-profile network remediations.
According to Michael Buratowski, vice president of cyber security services, for the past couple of years Fidelis has integrated XPS with the endpoint detection and response tool Carbon Black from Bit9 + Carbon Black.
Buratowski says on any given remediation they will be reviewing tens of thousands of endpoints, so they really need something that can speed up the forensics.
“We install Carbon Black on the endpoints, which lets us drill down without having to create a forensic image,” he explains.
Once Fidelis detects malware, it will sandbox the malicious code in a Fidelis sandbox and then Carbon Black runs an automated analysis.
“They just get us to remediation phase much faster,” Buratowski says. “Once Carbon Black sends back the report our remediation team can get to work and delete the malware.” -- S.Z.
IDT’s security team is constantly testing and evaluating products, so there may be 85 to 90 security technologies working on the network at any one time, a number he hopes to streamline in the next year as he sorts out redundant tools.
Ben-Oni has gotten good results from Resolution1 Security’s endpoint detection and response software. Resolution1 Security, which spun out from AccessData in January, correlates information from networks, endpoints and log management, and integrates with other systems to promote Continuous Automated Incident Response (CAIR). Once IDT gets an alert from its Palo Alto or FireEye firewalls, the information gets sent to Splunk, a log management system based on machine learning that evaluates the alert based on past history.
If Splunk determines that it’s a high-fidelity event, the code is sent to the Resolution1platform, which does a full threat analysis. If Resolution1 confirms it’s an advanced persistent threat or other high-fidelity event, IDT uses a custom-built tool to reimage the infected machine.
“We often have 60 investigations in a day and without automation it can take several hours to do just one, so there’s no way our staff could handle that,” Ben-Oni says. “The tools are simply much faster than people, so they are a big help.”
All-in-one endpoint security
Not all organizations are large federal agencies or a company the size and scope of IDT. Save Mart Supermarkets in Modesto, Calif., operates 220 stores (formerly Albertsons) in the San Francisco Bay Area and Northern Nevada.
Stephen Molina, information security administrator, says he doesn’t have the time or staff to test dozens of tools. He opted for AlienVault’s Unified Security Management appliance, which he says has all the security tools he needs built-in. The platform includes a server, sensors and logger, and it has access to threat intelligence from AlienVault’s Labs.
“We use it as a log aggregator, and it also does vulnerability scans,” he says, adding that AlienVault also includes an intrusion detection and prevention system.
“I can view the raw logs or alarms, and based on the information from the system can determine if it’s a false positive or not, and worth following up,” Molina says. “Once it does a vulnerability scan, it will correlate it against potential attacks, and if it’s, in fact, malware I can respond.”
Molina says when they locate a malware-infected system they replace it with a golden image. They use AlienVault primarily to protect Save Mart’s mission-critical point-of-sale systems in the stores.
“There’s no question this is a big time-saver for us,” he says. “If we weren’t using AlienVault, we’d have to run at least four or five other tools and appliances.”
No wonder there’s a looming sense that the bad guys are winning. It’s clear that security teams are looking for better ways to secure networks and endpoints, and reduce the time it takes to remediate an overwhelming number of security events.
Some organizations, like OPM and IDT, are fighting cyberthreats with a variety of network and endpoint tools, hoping over time to find the right mix. While many enterprises are testing out a variety of tools, what they really want are tools that can automate time-consuming manual tasks.
Others -- Save Mart Stores, for example -- realize they have to fight cyberthreats but can’t afford to test dozens of advanced threat detection and incident response tools.
Both approaches have merit. InfoSec professionals have to evaluate what’s best for their organizations and make the best judgment at the time -- knowing full well that today’s security approach may become tomorrow’s liability. It’s not always fun, but it’s the world we live in and why cybersecurity is such a challenge.
About the author
Steve Zurier is a freelance technology journalist based in Columbia, Md., with more than 30 years of journalism and publishing experience. Zurier previously worked as features editor at Government Computer News and InternetWeek.