In a perfect world (and I am dreaming "out loud" here), I would use an automated tool that would drive lots of different tests against a patched server and overnight it would give me a report stating that the results matched some predefined criteria allowing me to decide if the patch is safe, i.e. does not create other problems. Does such a tool exist?
Unfortunately, recent history makes it so that the best practice about patches is to install it as soon as possible, and the IS staffs are negligent if they delay. If you look at security problems having a window of vulnerability, delaying the install of a patch merely makes you more vulnerable. The problems they fix are so frightening that delay is bad.
You are right, in a perfect world, there would be a set of tests you can run against a server or workstation to make sure that things work properly. However, we don't get told what's in those patches, and so it is hard to know what to test. Often, the description of the problem is intentionally vague so as not to make it utterly obvious how to write an exploit program that can compromise machines during the time that IS groups are rolling out the fix.
In the case where a security fix is a browser fix, you are balancing the browser not working against one of your 10,000 users reading the wrong Web page that infects their machine. Pick your poison.
There aren't automated tools that can tell you if the patch is safe. It would be nice if there were. But alas, vendor software is not very good, and there's a reason they're creating the patch. Some Microsoft people I know told me that it costs them US$100,000 to release a patch. They don't do it because they want to. They do it because the problem is embarrassing enough to spend $100K to deliver a fix.
For more information on this topic, visit these other searchSecurity resources:
News & Analysis: Keeping up with patchwork near impossible
Tech Tip: Managing the patchwork mess
This was first published in February 2002