This article can also be found in the Premium Editorial Download "Information Security magazine: With SSL VPNs on the offense, will IPSec VPNs eventually be benched?."
Download it now to read this article plus other related content.
|Scheduled Releases? No thanks.|
After a patch-free March, Microsoft released a mountain of patches in April to fix 18 security holes. This pushed many IT shops into overdrive.
But, for Ben Marshall and Bill Arrington, not even the ugliest patching cycle can compare to their daily grind of two years ago--before Microsoft went to a monthly patch release and rolled out Software Update Services (SUS), a free Web application for Windows XP/2000/2003 that manages the prioritization and deployment of security updates.
Back then, patches arrived without warning. Marshall, Arrington and the rest of the network support team at Children's Hospital Boston would work late into the night, scrambling to figure out which patches affected their 200-plus servers and 4,000 workstations spread across the seven-building campus. "Before SUS, we'd manually update every workstation," Marshall says. "There are nine of us to manage desktops and servers, and we'd start at night and go until we were done, going building to building. We were always behind."
"We're much closer to the 9-to-5 life these days," Arring- ton says. "Now, we can set our patching schedule ahead of time and have people in place to test the patches. And the process is a lot more automated."
From Microsoft's perspective, the thinking behind Patch Tuesday was that a monthly rollout would improve both the patching process and the quality of the security fixes.
"If a product team released a patch on a Wednesday afternoon, an hour after it came out of a QA lab, it was shipped in an ad hoc manner. The only rule was there would be no shipments on Fridays," says Eric Schultze, chief security architect with Shavlik Technologies, a patch management software vendor, and a former program manager with Microsoft's Security Response Center (MSRC). "From the customer's perspective, they never knew when a patch was coming out. We would always catch them off guard."
Before, and well into the era of Trustworthy Comput-ing, Microsoft's patches were buggy and often broke other applications. Bugs in patches would only be discovered shortly after release, resulting in one, two, three and sometimes four iterations of the same patch. The January 2003 Slammer worm exploited a SQL vulnerability for which four patches had been released over the preceding six months.
"There was no chance to sit on it and evaluate the patch," Schultze says.
Microsoft moved to weekly releases before settling on the monthly update in October 2003.
"Most of the feedback we heard from customers was around planning. They told us they needed predictability to plan ahead," says Stephen Toulouse, security program manager at MSRC. "When we released security updates as soon as they were released from the testing phase, we got strong feedback that it was just not manageable."
April 12 brought five critical Microsoft Windows, Inter- net Explorer, Exchange, MSN Messenger and Word fixes, as well as three patches rated important for Windows. Contrarians would say Microsoft sat on those critical fixes for at least a month, extending the time enterprises were exposed to potential exploits. However, Microsoft will immediately release critical patches if an exploit is discovered in the wild--the last was a cumulative fix for IE released December 2004.
The company has also begun a third-party beta testing program, giving patches to select enterprises and government agencies 30 days in advance to test for conflicts and problems.
"Moving to a monthly release cycle allows our product teams to rigorously test the patches, and, in the end, that results in quality updates," says Toulouse. "We measure success by the increase in customers deploying updates and the growth in use of Windows Update and automatic updates."
Automation Is Key
For companies like BOC Edwards, Patch Tuesday is convenient, but not necessary.
"Patch Tuesday makes it easier for us to focus on a specific date. It gives us time to review a patch and its impact on us. The benefit, though, is not huge," says Dick Stewart, BOC's director of information services. "It does make it more likely that we won't overlook a vulnerability, but the automation is far and away more important to us."
BOC Edwards deploys patches to servers and workstations--except for some finicky boxes that require a manual touch--via PatchLink's PATCHLINK UPDATE. The automation provided by the patch management software not only pushes patches after a few clicks, but facilitates their prioritization. Once a software vendor pushes patches to BOC, managers scroll through the fixes to determine their impact on the company's environment and how many workstations and servers will be affected. Then, managers push it to two separate test environments, Alpha and Beta.
Alpha is made up of 10 machines that replicate the company's baseline system images. Patches are applied to determine if there are compatibility issues. Once they are confident, Stewart's teams push the patches through Beta, which are live production systems used by the information management team.
There, patches are tested more rigorously before being installed on production machines.
Smaller sites are patched first and monitored closely before patches are added to the baseline system images. If that goes smoothly, 600 machines are patched every two hours. The entire BOC network is patched in one night.
"We aren't quick to throw it on," Stewart says. "We won't patch on Day 1, then find out we shot ourselves in the foot. It could be two weeks until we're finished testing, unless it's a critical vulnerability."
At Children's Hospital, automation alleviates the running from one workstation to the next trying to figure out which machines need patching and which applications those patches are breaking. Children's Mar-shall uses SUS to prioritize, approve and push patches to the test lab. If a problem patch is flagged, he can click a few buttons to hold it while deploying the rest of the updates throughout the hospital's infrastructure. Once compatibility issues are resolved, he can quickly program SUS to complete the updates.
Most server updates are automated as well. Children's Arrington uses Ecora's Ecora Patch Manager to push updates to 10 servers at a time. Other servers, including Microsoft Exchange, are updated manually.
This was first published in May 2005