This article can also be found in the Premium Editorial Download "Information Security magazine: With SSL VPNs on the offense, will IPSec VPNs eventually be benched?."
Download it now to read this article plus other related content.
|A Pitch for Patching|
Out-of-cycle patch releases can still plunge IT teams into chaos.
"When [Microsoft] issues an out-of-cycle patch, it's always critical," says Children's Arrington. "The department testers aren't in place, so emergency notification goes out to all hands.
"If it's a Friday, we have a meeting to figure out if it's something that can wait until Monday. If not, we make plans for who will have to do what," he adds.
The last surprise was December's cumulative fix for critical IE vulnerabilities. That time, the staff chose to act immediately.
"When something is released out of cycle, you know it's big," Arrington says. "Under the old system, when patches could come at any time, it was much harder to determine what was critical, important or moderate."
Problems can also develop within the normal cycle.
It took a week before Children's Marshall was able to deploy April's patches. The process started smoothly enough, but then the lab ran into trouble with MS05-023, a critical fix for flaws in Microsoft Word and Office that would allow an attacker to take over vulnerable machines.
"It turned out that machines running Office without SP1 weren't compatible," Marshall says. With exploit code floating around cyberspace and worms like Mytob in the wild, the team had to decide whether to hold back all the patches and isolate problem machines or just delay deploying the one patch. This time, they chose to isolate the 67 incompatible machines.
"Those 67 users won't know the difference, and once their computers are updated to include SP1, we can put them back on the SUS system," Marshall says.
Small companies don't have the resources of BOC Edwards or Children's Hospital. Some have to rely on Microsoft for automation via Windows Update Services (WUS) to patch workstations, while often manually patching servers; others isolate machines from the Inter-net and patch as time allows.
"If a patch is very critical, it's scary for a company like ours because of the testing involved," says the VP of information services of a small Pennsylvania manufacturer. "If there's a SQL 2000 patch, do I want to put that patch on an enterprise server? It has gotten so big and complex that we have gone more on faith. We test what we can, but we just don't have the resources, time, money or software to do total software testing."
The VP, who didn't want his name or company published, tests patches on his home systems for application compatibility. If all goes well, he begins deployment.
"I don't think we are that uncommon at all," he says of 120-workstation, 13-server shop. "The nature of business cycles up and down. The amount of money I spend is based on business rather than need. We're a smaller shop with no resources. There are too many different versions of everything, including hardware."
Another small shop, a manufacturer in Washington, runs its mission-critical Oracle applications on Solaris and Red Hat, running other apps on 11 Windows NT, 2000 and 2003 servers. Critical patches are also pulled off WUS for 175 desktops, but downtime is scheduled for Sunday afternoons following Patch Tuesday for manual patching of enterprise servers.
"We apply it to production and trust Microsoft," says an administrator who also wished to remain anonymous. "It's different with Oracle; we have test boxes, and we test every patch. We install it, launch all the features and functions, test it out, and then put it into production. The same with Linux; we put patches into test boxes, test them, and then put them into production.
"We have enough confidence with Microsoft," he says. "It costs a fortune to have mirrors of every box with custom apps."
This was first published in May 2005