Ezine

This article can also be found in the Premium Editorial Download "Information Security magazine: With SSL VPNs on the offense, will IPSec VPNs eventually be benched?."

Download it now to read this article plus other related content.

Scheduled Releases? No thanks.

    Requires Free Membership to View

Don't count Arup Nanda among those in favor of regularly scheduled patch releases from Oracle. The database software giant last August followed Microsoft's lead of releasing security fixes on schedule (once a quarter), but Nanda, the lead DBA with Starwood Hotels & Resorts Worldwide, says he doesn't need help prioritizing his responsibilities.

"Servers are built around business requirements. You need to have applications up and running, not built around patch releases," says Nanda, who worries that quarterly patch releases extend the time vulnerable servers would be susceptible to attack.

His consternation was reinforced last September when Oracle released Alert 68 Rev. 2, which addressed security issues in the database server and listener--but lacked precious details about the vulnerabilities.

"If it falls into the wrong hands, a hacker could exploit it. But as a DBA, how do I go to management and ask for downtime to apply a patch when I don't know what the threat is? We want some kind of analysis from Oracle, and we're not getting it," Nanda says.

Oracle CSO Mary Ann Davidson acknowledges the rift over how much detail to provide in alerts.

"Oracle's position is to rely on customers to determine the right amount of information we need to provide so they can adequately gauge risk of patching versus not patching," Davidson says. "Given that the time to exploit has dropped dramatically, you can make a good case that providing too much detail only aids hackers by providing a road map for quicker exploits. Customers don't need proof-of-concept code to be able to gauge risk to their enterprises."

Nanda says he applies and tests Oracle patches in development, but the constantly flowing development cycle hinders how thorough that testing can be. Patches are deployed on a low-activity environment and monitored for compatibility issues. They're then pushed into production during scheduled Sunday night downtime.

"We take down the application, apply the patches and pray they don't break anything," Nanda says. His team of DBAs manages 50 production servers and 200 overall, including development and integration servers, each of which must be patched.

--Michael S. Mimoso

Controlled Chaos
After a patch-free March, Microsoft released a mountain of patches in April to fix 18 security holes. This pushed many IT shops into overdrive.

But, for Ben Marshall and Bill Arrington, not even the ugliest patching cycle can compare to their daily grind of two years ago--before Microsoft went to a monthly patch release and rolled out Software Update Services (SUS), a free Web application for Windows XP/2000/2003 that manages the prioritization and deployment of security updates.

Back then, patches arrived without warning. Marshall, Arrington and the rest of the network support team at Children's Hospital Boston would work late into the night, scrambling to figure out which patches affected their 200-plus servers and 4,000 workstations spread across the seven-building campus. "Before SUS, we'd manually update every workstation," Marshall says. "There are nine of us to manage desktops and servers, and we'd start at night and go until we were done, going building to building. We were always behind."

"We're much closer to the 9-to-5 life these days," Arring- ton says. "Now, we can set our patching schedule ahead of time and have people in place to test the patches. And the process is a lot more automated."

From Microsoft's perspective, the thinking behind Patch Tuesday was that a monthly rollout would improve both the patching process and the quality of the security fixes.

"If a product team released a patch on a Wednesday afternoon, an hour after it came out of a QA lab, it was shipped in an ad hoc manner. The only rule was there would be no shipments on Fridays," says Eric Schultze, chief security architect with Shavlik Technologies, a patch management software vendor, and a former program manager with Microsoft's Security Response Center (MSRC). "From the customer's perspective, they never knew when a patch was coming out. We would always catch them off guard."

Before, and well into the era of Trustworthy Comput-ing, Microsoft's patches were buggy and often broke other applications. Bugs in patches would only be discovered shortly after release, resulting in one, two, three and sometimes four iterations of the same patch. The January 2003 Slammer worm exploited a SQL vulnerability for which four patches had been released over the preceding six months.

"There was no chance to sit on it and evaluate the patch," Schultze says.

Microsoft moved to weekly releases before settling on the monthly update in October 2003.

"Most of the feedback we heard from customers was around planning. They told us they needed predictability to plan ahead," says Stephen Toulouse, security program manager at MSRC. "When we released security updates as soon as they were released from the testing phase, we got strong feedback that it was just not manageable."

April 12 brought five critical Microsoft Windows, Inter- net Explorer, Exchange, MSN Messenger and Word fixes, as well as three patches rated important for Windows. Contrarians would say Microsoft sat on those critical fixes for at least a month, extending the time enterprises were exposed to potential exploits. However, Microsoft will immediately release critical patches if an exploit is discovered in the wild--the last was a cumulative fix for IE released December 2004.

The company has also begun a third-party beta testing program, giving patches to select enterprises and government agencies 30 days in advance to test for conflicts and problems.

"Moving to a monthly release cycle allows our product teams to rigorously test the patches, and, in the end, that results in quality updates," says Toulouse. "We measure success by the increase in customers deploying updates and the growth in use of Windows Update and automatic updates."

Automation Is Key
For companies like BOC Edwards, Patch Tuesday is convenient, but not necessary.

"Patch Tuesday makes it easier for us to focus on a specific date. It gives us time to review a patch and its impact on us. The benefit, though, is not huge," says Dick Stewart, BOC's director of information services. "It does make it more likely that we won't overlook a vulnerability, but the automation is far and away more important to us."

BOC Edwards deploys patches to servers and workstations--except for some finicky boxes that require a manual touch--via PatchLink's PATCHLINK UPDATE. The automation provided by the patch management software not only pushes patches after a few clicks, but facilitates their prioritization. Once a software vendor pushes patches to BOC, managers scroll through the fixes to determine their impact on the company's environment and how many workstations and servers will be affected. Then, managers push it to two separate test environments, Alpha and Beta.

Alpha is made up of 10 machines that replicate the company's baseline system images. Patches are applied to determine if there are compatibility issues. Once they are confident, Stewart's teams push the patches through Beta, which are live production systems used by the information management team.

There, patches are tested more rigorously before being installed on production machines.

Smaller sites are patched first and monitored closely before patches are added to the baseline system images. If that goes smoothly, 600 machines are patched every two hours. The entire BOC network is patched in one night.

"We aren't quick to throw it on," Stewart says. "We won't patch on Day 1, then find out we shot ourselves in the foot. It could be two weeks until we're finished testing, unless it's a critical vulnerability."

At Children's Hospital, automation alleviates the running from one workstation to the next trying to figure out which machines need patching and which applications those patches are breaking. Children's Mar-shall uses SUS to prioritize, approve and push patches to the test lab. If a problem patch is flagged, he can click a few buttons to hold it while deploying the rest of the updates throughout the hospital's infrastructure. Once compatibility issues are resolved, he can quickly program SUS to complete the updates.

Most server updates are automated as well. Children's Arrington uses Ecora's Ecora Patch Manager to push updates to 10 servers at a time. Other servers, including Microsoft Exchange, are updated manually.

This was first published in May 2005

There are Comments. Add yours.

 
TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: