Lawrence M. Walsh
In previous worm outbreaks, enterprises had months to test and apply patches. Blaster stole that luxury, giving network and security mangers a scant four weeks to patch the critical Windows DCOM-RPC vulnerability.
There's little data to define a trend, but many in the security community say that the window for patching systems against publicly announced exploits is getting dramatically shorter.
"It's definitely starting to come down," says Andy Ellis, chief security architect at Akamai, a global Web content distribution service. "If you want to write a virus today, all you have to do is take existing code, drop in your exploit, and make architectural tweaks."
Worms are usually preceded by ample warning, which gives enterprises more than enough time to patch or secure their systems. SQL Slammer appeared last January, eight months after the vulnerability was announced and a patch released. In 2001, Nimda was preceded by a patch, five months of warnings and the Code Red worm, which essentially exploited the same IIS vulnerability.
In contrast, the DCOM vulnerability and patch were announced almost simultaneously in mid-July with the publishing of the exploit code. Blaster appeared August 12, just as enterprises were implementing their patching program.
"What we're seeing is if you don't already have a defense in place, you won't have any time to react anymore," says Kris Zupan, CEO of e-DMZ Security, a co-managed service provider. "It's no longer 'shame on the sysadmin' for not applying patches that are six or eight months old."
Enterprises are now facing an alarming prospect: repairing systems without testing the patches or else risk leaving critical systems exposed to exploits.
"A couple of weeks to test patches and put out a deployment plan isn't unreasonable," says Eric Schultze, executive director of product research and development at security tools vendor Shavlik Technologies. "If worms come out faster than that, major corporations are going to have a real problem."
One organization looking at the need for faster patch management is the Internal Revenue Service (IRS), which narrowly averted a major Blaster infection.
The IRS wasted no time responding to the critical RPC vulnerability advisory, obtaining the patch as soon as it was available. Following best practices, it tested the patch for interoperability and conflict issues. Then, it implemented a rollout plan to first patch its 5,000 Windows servers, after which would come its more than 125,000 Win2K and XP workstations.
"In an infrastructure this large, we have to balance the usability of the computer systems against their vulnerability," says Jim Kennedy, an IRS program manager. "When a patch like this comes out, the first thing we had to do is testing, so we could at least know the impact on our existing application infrastructure."
Blaster -- aka Lovsan -- interrupted that carefully crafted plan. By the time Kennedy discovered the worm on his network, it had already infected hundreds of machines. The servers had been patched, but tens of thousands of workstations were still vulnerable.
Leveraging Tivoli management infrastructure and using Symantec cleanup tools, IRS network and security teams -- numbering between 40 and 50 people -- patched more than 50,000 machines in nine hours. The rest were patched at a rate of 10,000 a day.
"I think we're on totally new ground," says Kennedy. "This new trend means we're going to have to react faster. The next time Microsoft releases a patch, we will apply that patch with a greater sense of urgency."
Hindering the patching process is the unreliability of patches. Enterprises can't blindly apply untested patches because they could open new security holes or break mission critical applications.
Slammer was successful because enterprises had a difficult time applying a series of faulty patches issued by Microsoft. The first three patches either didn't fix the vulnerability or undid previous security fixes.
Microsoft tested the DCOM patch prior to its release, but most enterprises still followed best practices that dictate testing and phased rollout.
"One of the things we're doing is working to improve all aspects of the patching process, including making it easier to roll the patches out across an organization. It's a work in progress," says Steve Lipner, Microsoft's director of security engineering strategies.
Be prepared: Lock down
Not all agree the patching window is closing, or that it needs to exist at all.
Worms appear at different intervals compared to vulnerabilities being discovered. The greatest concern is the dreaded zero-day exploit, the appearance of a worm that exploits a previously unpublicized/unknown vulnerability. On top of it all are the scores of vulnerabilities that are rarely exploited but still require patching.
"I'm not sure that I can make a real clear projection on what's happening out there," Lipner says. "There's a small number of points to try to draw a curve through."
Rather than worrying about patches, some say the answer resides in the basic network architecture, defense-in-depth security strategies and old-fashioned vigilance.
For Blaster, that would mean hardening Windows machines by turning off DCOM -- a noncritical service in most implementations. This would have bought enterprises valuable time to patch those few machines that require the service, and then roll out an organization-wide patch deployment.
"The final strategy is going to involve patch management at the OS level, more network defenses, network segmentation that will provide protection even when you're unaware of an exploit," says e-DMZ Security's Zupan. "We have to think this way because vulnerabilities and worms are coming out faster than the patches can be generated and distributed."