Information Security

Defending the digital infrastructure


Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Patch Tuesday

Microsoft's regular patch release cycle is a combination of information, process and automated tools that makes updates go more smoothly.

Microsoft's regular patch release cycle is a combination of information, process and automated tools that makes updates go more smoothly.

Microsoft intended its monthly patch bundles to help enterprises organize their vulnerability management efforts. But if you ask security managers what really makes a difference, they'll tell you it's automating those tricky patch deployments.

"Patch Tuesday," the second Tuesday of every month, usually delivers a heavy dose of security fixes of varied criticality that can challenge even the heartiest IT staff.

"When I was writing scripts, it would take me two hours to write just one, because you have to do research on a patch, download it and deploy it to one machine, then monitor the logs for problems," says Keith Seavey, infrastructure analyst for BOC Edwards, a Wilmington, Mass.-based semiconductor manufacturer with 4,000 seats and 250 servers at 68 sites around the world.

"To patch the whole U.S. environment this way would take 40 hours--times-two for Europe and Asia. Now it takes five minutes," he adds. "We're far more likely to patch, when before we were putting on only critical patches that applied to our environment."

But is Patch Tuesday a blessing or a curse for enterprises? To find out, Information Security and visited several organizations of various sizes on Patch Tuesday and the days that followed to see the benefits, challenges and shortcomings of the scheduled patch release cycle. What we found is that the combination of predictable patch release dates, better testing and automated patch management tools is paying real dividends.

Patch Tuesday: A Weeklong Day
Patch Tuesday is the day Microsoft releases patches each month. But, as is the case at Children's Hospital Boston, Patch Tuesday often extends into a week-plus of activity.

Here's the timeline of Children's typical patching cycle.

Scheduled Releases? No thanks.
Don't count Arup Nanda among those in favor of regularly scheduled patch releases from Oracle. The database software giant last August followed Microsoft's lead of releasing security fixes on schedule (once a quarter), but Nanda, the lead DBA with Starwood Hotels & Resorts Worldwide, says he doesn't need help prioritizing his responsibilities.

"Servers are built around business requirements. You need to have applications up and running, not built around patch releases," says Nanda, who worries that quarterly patch releases extend the time vulnerable servers would be susceptible to attack.

His consternation was reinforced last September when Oracle released Alert 68 Rev. 2, which addressed security issues in the database server and listener--but lacked precious details about the vulnerabilities.

"If it falls into the wrong hands, a hacker could exploit it. But as a DBA, how do I go to management and ask for downtime to apply a patch when I don't know what the threat is? We want some kind of analysis from Oracle, and we're not getting it," Nanda says.

Oracle CSO Mary Ann Davidson acknowledges the rift over how much detail to provide in alerts.

"Oracle's position is to rely on customers to determine the right amount of information we need to provide so they can adequately gauge risk of patching versus not patching," Davidson says. "Given that the time to exploit has dropped dramatically, you can make a good case that providing too much detail only aids hackers by providing a road map for quicker exploits. Customers don't need proof-of-concept code to be able to gauge risk to their enterprises."

Nanda says he applies and tests Oracle patches in development, but the constantly flowing development cycle hinders how thorough that testing can be. Patches are deployed on a low-activity environment and monitored for compatibility issues. They're then pushed into production during scheduled Sunday night downtime.

"We take down the application, apply the patches and pray they don't break anything," Nanda says. His team of DBAs manages 50 production servers and 200 overall, including development and integration servers, each of which must be patched.

--Michael S. Mimoso

Controlled Chaos
After a patch-free March, Microsoft released a mountain of patches in April to fix 18 security holes. This pushed many IT shops into overdrive.

But, for Ben Marshall and Bill Arrington, not even the ugliest patching cycle can compare to their daily grind of two years ago--before Microsoft went to a monthly patch release and rolled out Software Update Services (SUS), a free Web application for Windows XP/2000/2003 that manages the prioritization and deployment of security updates.

Back then, patches arrived without warning. Marshall, Arrington and the rest of the network support team at Children's Hospital Boston would work late into the night, scrambling to figure out which patches affected their 200-plus servers and 4,000 workstations spread across the seven-building campus. "Before SUS, we'd manually update every workstation," Marshall says. "There are nine of us to manage desktops and servers, and we'd start at night and go until we were done, going building to building. We were always behind."

"We're much closer to the 9-to-5 life these days," Arring- ton says. "Now, we can set our patching schedule ahead of time and have people in place to test the patches. And the process is a lot more automated."

From Microsoft's perspective, the thinking behind Patch Tuesday was that a monthly rollout would improve both the patching process and the quality of the security fixes.

"If a product team released a patch on a Wednesday afternoon, an hour after it came out of a QA lab, it was shipped in an ad hoc manner. The only rule was there would be no shipments on Fridays," says Eric Schultze, chief security architect with Shavlik Technologies, a patch management software vendor, and a former program manager with Microsoft's Security Response Center (MSRC). "From the customer's perspective, they never knew when a patch was coming out. We would always catch them off guard."

Before, and well into the era of Trustworthy Comput-ing, Microsoft's patches were buggy and often broke other applications. Bugs in patches would only be discovered shortly after release, resulting in one, two, three and sometimes four iterations of the same patch. The January 2003 Slammer worm exploited a SQL vulnerability for which four patches had been released over the preceding six months.

"There was no chance to sit on it and evaluate the patch," Schultze says.

Microsoft moved to weekly releases before settling on the monthly update in October 2003.

"Most of the feedback we heard from customers was around planning. They told us they needed predictability to plan ahead," says Stephen Toulouse, security program manager at MSRC. "When we released security updates as soon as they were released from the testing phase, we got strong feedback that it was just not manageable."

April 12 brought five critical Microsoft Windows, Inter- net Explorer, Exchange, MSN Messenger and Word fixes, as well as three patches rated important for Windows. Contrarians would say Microsoft sat on those critical fixes for at least a month, extending the time enterprises were exposed to potential exploits. However, Microsoft will immediately release critical patches if an exploit is discovered in the wild--the last was a cumulative fix for IE released December 2004.

The company has also begun a third-party beta testing program, giving patches to select enterprises and government agencies 30 days in advance to test for conflicts and problems.

"Moving to a monthly release cycle allows our product teams to rigorously test the patches, and, in the end, that results in quality updates," says Toulouse. "We measure success by the increase in customers deploying updates and the growth in use of Windows Update and automatic updates."

Automation Is Key
For companies like BOC Edwards, Patch Tuesday is convenient, but not necessary.

"Patch Tuesday makes it easier for us to focus on a specific date. It gives us time to review a patch and its impact on us. The benefit, though, is not huge," says Dick Stewart, BOC's director of information services. "It does make it more likely that we won't overlook a vulnerability, but the automation is far and away more important to us."

BOC Edwards deploys patches to servers and workstations--except for some finicky boxes that require a manual touch--via PatchLink's PATCHLINK UPDATE. The automation provided by the patch management software not only pushes patches after a few clicks, but facilitates their prioritization. Once a software vendor pushes patches to BOC, managers scroll through the fixes to determine their impact on the company's environment and how many workstations and servers will be affected. Then, managers push it to two separate test environments, Alpha and Beta.

Alpha is made up of 10 machines that replicate the company's baseline system images. Patches are applied to determine if there are compatibility issues. Once they are confident, Stewart's teams push the patches through Beta, which are live production systems used by the information management team.

There, patches are tested more rigorously before being installed on production machines.

Smaller sites are patched first and monitored closely before patches are added to the baseline system images. If that goes smoothly, 600 machines are patched every two hours. The entire BOC network is patched in one night.

"We aren't quick to throw it on," Stewart says. "We won't patch on Day 1, then find out we shot ourselves in the foot. It could be two weeks until we're finished testing, unless it's a critical vulnerability."

At Children's Hospital, automation alleviates the running from one workstation to the next trying to figure out which machines need patching and which applications those patches are breaking. Children's Mar-shall uses SUS to prioritize, approve and push patches to the test lab. If a problem patch is flagged, he can click a few buttons to hold it while deploying the rest of the updates throughout the hospital's infrastructure. Once compatibility issues are resolved, he can quickly program SUS to complete the updates.

Most server updates are automated as well. Children's Arrington uses Ecora's Ecora Patch Manager to push updates to 10 servers at a time. Other servers, including Microsoft Exchange, are updated manually.

A Pitch for Patching
We asked seven patch vendors to design a system for a hypothetical enterprise. See what they proposed.

By: Neil Roiter

Patch management can bleed organizations of time and resources. Delay exposes critical assets to potential compromise, while inadequately tested patches can break apps and systems. The cycle of discovery, testing, remediation, verification and re-remediation is a major drain on SMBs and Fortune 1000 companies alike.

Automated patch/configuration management tools are designed to relieve these pain points, allowing organizations to enforce security polices and processes quickly and cost-effectively. Companies can choose between various commercial products and services, but how do they pick the right product for their environment?

Assuming that companies buy solutions and not just cool feature sets, we invited vendors to respond to a request for proposal (RFP) from a hypothetical company suffering all the miseries associated with patching a highly distributed environment.

Information Security assembled a panel of four infosecurity experts to evaluate the seven responses that best addressed the specific requirements detailed in our scenario--from BigFix, Citadel Security Software, Configure-soft, Everdream, PatchLink, St. Bernard Software and Shavlik Technologies.

The responses were revealing, showing how different vendors address customer needs on a conceptual and planning level. Some were polished, others creative, and a few just disappointing. A complete analysis is available online; the following is a synopsis:

  • BigFix provided a clear, detailed explanation of its capabilities, addressed directly to our requirements. It was the only vendor that provided a deployment diagram. Its highly flexible agent-based BigFix Enterprise Server (BES) automates vulnerability assessment and remediation for LAN-based and mobile devices. Its proposed Relay Server architecture addressed one of our main concerns: patching small satellite offices in three countries.
  • Citadel Software Security didn't do a very good job explaining its Hercules product architecture technology and how it could be deployed to address our company's needs. Although it answered the requirements, we were left with a vague picture and too much brochure-type material about the company.
  • Configuresoft's response emphasized the Enterprise Configuration Manager (ECM) and Security Update Manager (SUM) suite's strength as a configuration management and automated patch management product, turning the issue, correctly, into one of overall vulnerability management. However, it didn't provide a real plan for our company. Configuresoft's agents offer flexibility for setting policy and automating remediation.
  • Everdream's Everdream Patch Management is the only managed service among the respondents, offering the usual trade-offs of offloading the work versus loss of control. Its responses were generally of the "we can do this" variety, lacking specific recommendations.
  • PatchLink spelled out specific architecture recommendations and clearly explained how PATCHLINK UPDATE works and why it was well suited to meet our requirements. Its flexible agents, verification technology and Distribution Points met our small office needs. It also described the most thorough patch-testing regimen among the vendors.
  • St. Bernard Software did a capable job explaining how its UpdateEXPERT works, but it offered few specifics that addressed our company's needs. Its option for agent-based and agentless technology offers great flexibility for using agents on mobile devices and, possibly, in satellite offices.
  • Shavlik Technologies did an excellent job explaining the features of its product, HFNetchkPro, and offered two distinct deployment scenarios: central management and distributed management. However, it made no attempt to discuss its product in terms of our needs, leaving us to pick through the response to figure out what was relevant.

For more information and the full text of this article, visit

Neil Roiter is senior technology editor at Information Security.

Unwelcome Surprises
Out-of-cycle patch releases can still plunge IT teams into chaos.

"When [Microsoft] issues an out-of-cycle patch, it's always critical," says Children's Arrington. "The department testers aren't in place, so emergency notification goes out to all hands.

"If it's a Friday, we have a meeting to figure out if it's something that can wait until Monday. If not, we make plans for who will have to do what," he adds.

The last surprise was December's cumulative fix for critical IE vulnerabilities. That time, the staff chose to act immediately.

"When something is released out of cycle, you know it's big," Arrington says. "Under the old system, when patches could come at any time, it was much harder to determine what was critical, important or moderate."

Problems can also develop within the normal cycle.

It took a week before Children's Marshall was able to deploy April's patches. The process started smoothly enough, but then the lab ran into trouble with MS05-023, a critical fix for flaws in Microsoft Word and Office that would allow an attacker to take over vulnerable machines.

"It turned out that machines running Office without SP1 weren't compatible," Marshall says. With exploit code floating around cyberspace and worms like Mytob in the wild, the team had to decide whether to hold back all the patches and isolate problem machines or just delay deploying the one patch. This time, they chose to isolate the 67 incompatible machines.

"Those 67 users won't know the difference, and once their computers are updated to include SP1, we can put them back on the SUS system," Marshall says.

Size Matters
Small companies don't have the resources of BOC Edwards or Children's Hospital. Some have to rely on Microsoft for automation via Windows Update Services (WUS) to patch workstations, while often manually patching servers; others isolate machines from the Inter-net and patch as time allows.

"If a patch is very critical, it's scary for a company like ours because of the testing involved," says the VP of information services of a small Pennsylvania manufacturer. "If there's a SQL 2000 patch, do I want to put that patch on an enterprise server? It has gotten so big and complex that we have gone more on faith. We test what we can, but we just don't have the resources, time, money or software to do total software testing."

The VP, who didn't want his name or company published, tests patches on his home systems for application compatibility. If all goes well, he begins deployment.

"I don't think we are that uncommon at all," he says of 120-workstation, 13-server shop. "The nature of business cycles up and down. The amount of money I spend is based on business rather than need. We're a smaller shop with no resources. There are too many different versions of everything, including hardware."

Another small shop, a manufacturer in Washington, runs its mission-critical Oracle applications on Solaris and Red Hat, running other apps on 11 Windows NT, 2000 and 2003 servers. Critical patches are also pulled off WUS for 175 desktops, but downtime is scheduled for Sunday afternoons following Patch Tuesday for manual patching of enterprise servers.

"We apply it to production and trust Microsoft," says an administrator who also wished to remain anonymous. "It's different with Oracle; we have test boxes, and we test every patch. We install it, launch all the features and functions, test it out, and then put it into production. The same with Linux; we put patches into test boxes, test them, and then put them into production.

"We have enough confidence with Microsoft," he says. "It costs a fortune to have mirrors of every box with custom apps."

Article 2 of 13

Dig Deeper on Microsoft Patch Tuesday and patch management

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.

Get More Information Security

Access to all of our back issues View All