Published: 30 May 2005
Microsoft's regular patch release cycle is a combination of information, process and automated tools that makes updates go more smoothly.
Microsoft intended its monthly patch bundles to help enterprises organize their vulnerability management efforts. But if you ask security managers what really makes a difference, they'll tell you it's automating those tricky patch deployments.
"Patch Tuesday," the second Tuesday of every month, usually delivers a heavy dose of security fixes of varied criticality that can challenge even the heartiest IT staff.
"When I was writing scripts, it would take me two hours to write just one, because you have to do research on a patch, download it and deploy it to one machine, then monitor the logs for problems," says Keith Seavey, infrastructure analyst for BOC Edwards, a Wilmington, Mass.-based semiconductor manufacturer with 4,000 seats and 250 servers at 68 sites around the world.
"To patch the whole U.S. environment this way would take 40 hours--times-two for Europe and Asia. Now it takes five minutes," he adds. "We're far more likely to patch, when before we were putting on only critical patches that applied to our environment."
But is Patch Tuesday a blessing or a curse for enterprises? To find out, Information Security and SearchSecurity.com visited several organizations of various sizes on Patch Tuesday and the days that followed to see the benefits, challenges and shortcomings of the scheduled patch release cycle. What we found is that the combination of predictable patch release dates, better testing and automated patch management tools is paying real dividends.
|Scheduled Releases? No thanks.|
After a patch-free March, Microsoft released a mountain of patches in April to fix 18 security holes. This pushed many IT shops into overdrive.
But, for Ben Marshall and Bill Arrington, not even the ugliest patching cycle can compare to their daily grind of two years ago--before Microsoft went to a monthly patch release and rolled out Software Update Services (SUS), a free Web application for Windows XP/2000/2003 that manages the prioritization and deployment of security updates.
Back then, patches arrived without warning. Marshall, Arrington and the rest of the network support team at Children's Hospital Boston would work late into the night, scrambling to figure out which patches affected their 200-plus servers and 4,000 workstations spread across the seven-building campus. "Before SUS, we'd manually update every workstation," Marshall says. "There are nine of us to manage desktops and servers, and we'd start at night and go until we were done, going building to building. We were always behind."
"We're much closer to the 9-to-5 life these days," Arring- ton says. "Now, we can set our patching schedule ahead of time and have people in place to test the patches. And the process is a lot more automated."
From Microsoft's perspective, the thinking behind Patch Tuesday was that a monthly rollout would improve both the patching process and the quality of the security fixes.
"If a product team released a patch on a Wednesday afternoon, an hour after it came out of a QA lab, it was shipped in an ad hoc manner. The only rule was there would be no shipments on Fridays," says Eric Schultze, chief security architect with Shavlik Technologies, a patch management software vendor, and a former program manager with Microsoft's Security Response Center (MSRC). "From the customer's perspective, they never knew when a patch was coming out. We would always catch them off guard."
Before, and well into the era of Trustworthy Comput-ing, Microsoft's patches were buggy and often broke other applications. Bugs in patches would only be discovered shortly after release, resulting in one, two, three and sometimes four iterations of the same patch. The January 2003 Slammer worm exploited a SQL vulnerability for which four patches had been released over the preceding six months.
"There was no chance to sit on it and evaluate the patch," Schultze says.
Microsoft moved to weekly releases before settling on the monthly update in October 2003.
"Most of the feedback we heard from customers was around planning. They told us they needed predictability to plan ahead," says Stephen Toulouse, security program manager at MSRC. "When we released security updates as soon as they were released from the testing phase, we got strong feedback that it was just not manageable."
April 12 brought five critical Microsoft Windows, Inter- net Explorer, Exchange, MSN Messenger and Word fixes, as well as three patches rated important for Windows. Contrarians would say Microsoft sat on those critical fixes for at least a month, extending the time enterprises were exposed to potential exploits. However, Microsoft will immediately release critical patches if an exploit is discovered in the wild--the last was a cumulative fix for IE released December 2004.
The company has also begun a third-party beta testing program, giving patches to select enterprises and government agencies 30 days in advance to test for conflicts and problems.
"Moving to a monthly release cycle allows our product teams to rigorously test the patches, and, in the end, that results in quality updates," says Toulouse. "We measure success by the increase in customers deploying updates and the growth in use of Windows Update and automatic updates."
Automation Is Key
For companies like BOC Edwards, Patch Tuesday is convenient, but not necessary.
"Patch Tuesday makes it easier for us to focus on a specific date. It gives us time to review a patch and its impact on us. The benefit, though, is not huge," says Dick Stewart, BOC's director of information services. "It does make it more likely that we won't overlook a vulnerability, but the automation is far and away more important to us."
BOC Edwards deploys patches to servers and workstations--except for some finicky boxes that require a manual touch--via PatchLink's PATCHLINK UPDATE. The automation provided by the patch management software not only pushes patches after a few clicks, but facilitates their prioritization. Once a software vendor pushes patches to BOC, managers scroll through the fixes to determine their impact on the company's environment and how many workstations and servers will be affected. Then, managers push it to two separate test environments, Alpha and Beta.
Alpha is made up of 10 machines that replicate the company's baseline system images. Patches are applied to determine if there are compatibility issues. Once they are confident, Stewart's teams push the patches through Beta, which are live production systems used by the information management team.
There, patches are tested more rigorously before being installed on production machines.
Smaller sites are patched first and monitored closely before patches are added to the baseline system images. If that goes smoothly, 600 machines are patched every two hours. The entire BOC network is patched in one night.
"We aren't quick to throw it on," Stewart says. "We won't patch on Day 1, then find out we shot ourselves in the foot. It could be two weeks until we're finished testing, unless it's a critical vulnerability."
At Children's Hospital, automation alleviates the running from one workstation to the next trying to figure out which machines need patching and which applications those patches are breaking. Children's Mar-shall uses SUS to prioritize, approve and push patches to the test lab. If a problem patch is flagged, he can click a few buttons to hold it while deploying the rest of the updates throughout the hospital's infrastructure. Once compatibility issues are resolved, he can quickly program SUS to complete the updates.
Most server updates are automated as well. Children's Arrington uses Ecora's Ecora Patch Manager to push updates to 10 servers at a time. Other servers, including Microsoft Exchange, are updated manually.
|A Pitch for Patching|
Out-of-cycle patch releases can still plunge IT teams into chaos.
"When [Microsoft] issues an out-of-cycle patch, it's always critical," says Children's Arrington. "The department testers aren't in place, so emergency notification goes out to all hands.
"If it's a Friday, we have a meeting to figure out if it's something that can wait until Monday. If not, we make plans for who will have to do what," he adds.
The last surprise was December's cumulative fix for critical IE vulnerabilities. That time, the staff chose to act immediately.
"When something is released out of cycle, you know it's big," Arrington says. "Under the old system, when patches could come at any time, it was much harder to determine what was critical, important or moderate."
Problems can also develop within the normal cycle.
It took a week before Children's Marshall was able to deploy April's patches. The process started smoothly enough, but then the lab ran into trouble with MS05-023, a critical fix for flaws in Microsoft Word and Office that would allow an attacker to take over vulnerable machines.
"It turned out that machines running Office without SP1 weren't compatible," Marshall says. With exploit code floating around cyberspace and worms like Mytob in the wild, the team had to decide whether to hold back all the patches and isolate problem machines or just delay deploying the one patch. This time, they chose to isolate the 67 incompatible machines.
"Those 67 users won't know the difference, and once their computers are updated to include SP1, we can put them back on the SUS system," Marshall says.
Small companies don't have the resources of BOC Edwards or Children's Hospital. Some have to rely on Microsoft for automation via Windows Update Services (WUS) to patch workstations, while often manually patching servers; others isolate machines from the Inter-net and patch as time allows.
"If a patch is very critical, it's scary for a company like ours because of the testing involved," says the VP of information services of a small Pennsylvania manufacturer. "If there's a SQL 2000 patch, do I want to put that patch on an enterprise server? It has gotten so big and complex that we have gone more on faith. We test what we can, but we just don't have the resources, time, money or software to do total software testing."
The VP, who didn't want his name or company published, tests patches on his home systems for application compatibility. If all goes well, he begins deployment.
"I don't think we are that uncommon at all," he says of 120-workstation, 13-server shop. "The nature of business cycles up and down. The amount of money I spend is based on business rather than need. We're a smaller shop with no resources. There are too many different versions of everything, including hardware."
Another small shop, a manufacturer in Washington, runs its mission-critical Oracle applications on Solaris and Red Hat, running other apps on 11 Windows NT, 2000 and 2003 servers. Critical patches are also pulled off WUS for 175 desktops, but downtime is scheduled for Sunday afternoons following Patch Tuesday for manual patching of enterprise servers.
"We apply it to production and trust Microsoft," says an administrator who also wished to remain anonymous. "It's different with Oracle; we have test boxes, and we test every patch. We install it, launch all the features and functions, test it out, and then put it into production. The same with Linux; we put patches into test boxes, test them, and then put them into production.
"We have enough confidence with Microsoft," he says. "It costs a fortune to have mirrors of every box with custom apps."