Network administrators spend their days overwhelmed by the task of testing and deploying patches to vulnerable systems. It's not just the volume of patches that gets them down but, once patches have been deployed, they've got to worry that those fixes don't break other applications.
And all the while, piles of other demanding work grow exponentially.
It's a vicious circle, and there are no signs of that circle being broken.
"After the experience we had dealing with the RPC vulnerabilities and attacks [in August], patch management has become a priority for everyone I know in an organization," said Tina Bird, a security officer with the Information Technology Systems and Services organization at Stanford University.
Bird and other security luminaries, led primarily by Eric Schultze, chief security architect at Shavlik Technologies LLC and former program manager for Microsoft's Security Response Center, started the PatchManagement.org mailing list earlier this month, giving administrators a forum for discussing patch management tools and strategies. Other moderators include Jason Chan, a principal security architect for consultancy AtStake Inc., and Ben Laurie, a director at the Apache Software Foundation.
"Other people have to deal with political and technical issues and have different ideas on doing things," Bird said. "The way to solve problems is hands-on -- get a group talking together."
Naturally, Microsoft is dominating daily discussions, and many posters would say with good reason. Microsoft was at the heart of most headline security incidents in 2003, starting with the Slammer outbreak in January and continuing with the RPC holes and, in August, the Blaster worm, to recent problems with Exchange and Workstation services.
Absorbing all these blows is the tattered network administrator trying to figure out how to prioritize patches for testing and deployment, and trying to pinpoint where all his vulnerable machines may be.
"It comes down to what thing would I drop everything for, get up at 3 a.m. to fix right now -- those vulnerabilities in default installations that are accessible without the user doing anything," Bird said. "Especially if there's an exploit in the wild. That's what made RPC so fatal. The RPC stuff spread without anyone having to do anything."
Prior to this summer of vulnerabilities, admins were already swamped with patching perplexities. Bird and her security team at Stanford, for example, spend two and a half months working full time putting out RPC fires on the university's 40,000-node network.
"Prior to RPC, there was the general perception that too many patches were released, that the dependencies were too complicated and it was already too hard to deal with," Bird said. "Then we got whammied and, if we don't figure things out, we're going to spend the rest of our lives doing this manually."
Microsoft has slowed its patch-release cycle to a series of monthly updates, save for the occasional critical fix. But, as Bird said, that doesn't eliminate the problem of Redmond's ubiquitous and faulty software. The problem, meanwhile, has escalated as hackers have turned their attention to network services like RPC rather than solely attacking applications like Internet Explorer, she said.
"Well, if you want to bring the Internet to a crashing halt, the best way is to attack things that are listening by default," Bird said. "By and large, these are desktop machines [that are being exploited], and if you're running Linux, there are no ports open [unlike Windows]. You've got a major problem that network infrastructures [are] run by people that don't want this technology offering services [turned on by default]. And then, Microsoft's answer is to turn on the firewall."
The end result may be automated patch management, something that Bird said is happening already in many enterprises, despite resistance on many fronts from administrators unwilling to relinquish control of what is installed on their systems.
"This summer was really an eye-opener for a lot of people," Bird said. "I've spent most of my life monitoring the Internet for these things, then describing them to my users to protect them. I tell them enable Windows Update and check off the box that loads everything. Some worry about patches breaking other applications and don't feel comfortable -- and that's a rational concern. But if you say that, that means you want to read the bulletin [from Microsoft], find what's relative to your machines and take action."
FEEDBACK: How do you prioritize your patching?
Send your feedback to the SearchSecurity.com news team.