Published: 01 Dec 2003
Greg McGill is so tired of IT staffers haphazardly deploying broken patches on his network that he's considering a labor-intensive, highly regulated patch brokerage -- rather than an automated commercial solution -- to oversee vulnerability remediation.
"In a large enterprise, you need to guard against the zeal of sysadmins trying to keep their systems so up to date that they err in the wrong direction -- a problem we probably would not have predicted two years ago," says McGill, a network architect at Blue Cross/Blue Shield of Alabama.
That's because in the last two years -- and particularly the last 12 months -- the infosecurity community has been consumed with fixes and upgrades to seal software vulnerabilities. While the number of reported vulnerabilities actually decreased in 2003 from the previous year, the acute problems brought on by several major malware outbreaks makes this "The Year of the Patch."
No other area of infosecurity has created as many headlines -- and headaches -- as patch management and the consequences when vulnerabilities aren't addressed. And, making matters worse, virus writers are releasing exploits faster.
That's essentially what led to the severity of last January's SQL Slammer worm, which bored through networks and shut down operations, including air traffic control in some areas. Then, in August, came Blaster and Sobig.F, malware that caused $2 billion collectively in data and productivity losses worldwide, according to Computer Economics.
In all three outbreaks, a security patch had been released for the Microsoft software exploited. But the fix apparently wasn't widely applied, given the millions of machines that were quickly infected.
Vendors tried to assist by offering numerous patching tools. The projected 68 percent increase in revenue related to patch solutions this year indicates there are plenty of buyers, though that number represents a mere $17.9 million in total sales, according to Gartner Group.
The patch management market figures don't take into account the number of free and embedded patching features in many operating systems and applications. Further, enterprises frequently are writing custom applications to push out patches.
At the other end of the spectrum is growing pressure on software developers -- particularly Microsoft -- to create more secure code before a product's release.
"Operating system vendors have, over the last five or six years, focused on functionality first. Everybody wants to come out with a cool new tool, and if you're behind that curve, you look bad," explains Mark Mellis, a California-based network consultant with SystemsExperts. "Security hadn't impacted people until recently, but we've finally gotten to a point where it really does matter. The bad things you get in your e-mail or over your network connection now really do have an effect on your ability to do anything on your computer."
The number of incidents reported to the CERT Coordination Center is up this year -- 114,855 through Q3 compared to 82,094 for all of 2002. Vulnerability reports appear to be slightly decreasing, from 4,129 last year to 2,982 as of last quarter -- meaning 2003 is tracking to be the second worst year on record. Because CERT stats only run through September, it's difficult to say if 28 security alerts published thus far will surpass or fall short of the 41 issued in 2002 and 2001.
Still, reducing the number of security holes requires going to the source. "The key is making the software secure. We really have a hard time with that," Mellis says. "But, I actually think, Microsoft cares about security now."
Microsoft should. Just last month the software giant, which remains a popular target for virus writers because of its products' ubiquity, admitted security breaches led to a $768 million decline in "deferred revenue" -- multiyear contracts counted towards earnings over time. That was some $450 million lower than the company had expected for its first quarter.
Industry watchers speculate the revenue drop meant customers not only held off contracts in protest, but that IT money earmarked for new software was diverted to cover the cost of patching Windows systems.
"It's going to cost them business, and that's what gets Microsoft's attention," Mellis says, quickly noting patching isn't just "a Microsoft problem." Red Hat, which produces a commercial version of the Linux OS, issues patches almost daily, and Apple's Macintosh OS is far from patch-free.
4: The number of patches Microsoft released for the SQL vulnerability exploited by Slammer. The fourth patch, released a month before Slammer appeared, finally fixed the vulnerability. (Microsoft)
6: Months between the discovery of the SQL vulnerability and the appearance of the Slammer worm. (TruSecure/ICSA Labs)
26 Days: Time between the release of the patch for the RPC-DCOM Windows vulnerability and appearance of the Blaster worm. At one point, Blaster was infecting up to 2,500 computers per hour. (Symantec)
50: Percentage of all vulnerabilities left unaddressed 30 days after the release of software patches. (Qualys)
80: Percentage of exploits released within two months of a vulnerability's public disclosure. (Qualys)
$475,000: Median vulnerability remediation cost -- including patching -- per company. This includes hard, soft and productivity costs. (TruSecure/ICSA Labs)
To its credit, Microsoft has restructured its patch release program to make it easier on enterprises. Bulletins now come out monthly, rather than weekly, and can be found easily online.
Meantime, seasoned IT vets like Blue Cross/Blue Shields' McGill are creating their own solutions to patch management, which not only entail applying patches in a timely manner, but making sure they don't break other applications in the process. That's a problem that plagues small and large enterprises alike.
The idea of a "patch brokerage" -- which is still in the embryonic stage -- came as much from pleasing regulatory auditors as filling a marketplace void.
"Being an insurance company, we're up against a lot of different auditors, and the question they always have is, "Well, your different LAN admins put the various patches on, but how do you know it really went on?" explains McGill.
A difference between current solutions and a brokerage is the QA process that all patches go through before they're selectively deployed.
"One of the industry's problems is the idea that all you have to do is put a patch on. But we have so much custom code, especially large enterprises that are running code they wrote themselves," McGill says. "And that's one thing a product ought to be able to do -- broker that.
"I think it's time for a major change in the philosophy of patch management," McGill says.