One of the reasons some organizations fail to properly test is that they believe there is not enough time to test a patch prior to deploying it. Given the shortened patch management life cycle from when a vulnerability is discovered, to when a patch is released, to the time an exploit is running rampant over the Internet, this almost becomes a valid argument. However, most of these organizations made this claim when there were several months between the release of a patch and an actual exploit.
The organizations making this claim are usually considered to be stuck in reactive mode, as opposed to being proactive when it comes to patching their systems. They wait until an exploit has actually affected their organization before they begin to react. This method of addressing patches is chaotic to say the least, but becomes more chaotic when systems react negatively to a bad patch or misconfiguration that could have been caught with an effective testing process. Many hours of needless remediation then follow in determining what caused the problems when applying the patch to systems, fixing systems that were affected by early production release of the patch and trying to keep the remaining operational, but unpatched systems, from being exploited. An effective patch testing process can shorten the time to production release of patches while minimizing the chances that a patch will have an adverse impact on operations and allowing the organization to become more proactive in managing the business' exposure.
Along with lacking a patch rating system, organizations often fail to prioritize which systems should be tested first in their testing process. This can leave companies scrambling when an exploit hits as they test and deploy patches to production systems that had less of an exposure level. Guidelines should be created that serve a business in making sure that systems with higher exposure levels are addressed in proper sequence. Doing so can save valuable resource time and hopefully decrease downtime of nonpatched systems that are needlessly suffering from an exploit.
Others tend to excuse themselves from effective testing because they think they need an elaborate and expensive testing facility. Depending on the size and complexity of the environment, this may or may not be the case. Typically, organizations that are not just trying to make excuses can create an effective testing facility that might consist of a few pieces of additional hardware set up in an area that will not have its co-workers tripping over Ethernet cables. In addition, the money that proactive testing of patches will save in lost production system recovery time can be used to work toward a more elaborate environment at some time in the future, not to mention possibly saving jobs within IT.
Virtual machine software can also alleviate a portion of the traditional costs associated with loading test systems. Given ample RAM, disk space, and processor capacity, virtual machine software can eliminate the need for large quantities of hardware that take up expensive office space, increase environmental costs for cooling, and other drawbacks that are normally part of hosting a dedicated facility. Moreover, because of virtual machine capabilities, the expensive resource time that was necessary for setting up replicas of production systems and reconfiguration of systems can be greatly reduced.
This chapter explores these issues as well as provides more detail on how to effectively address each of these common problems.
Download Chapter 8, Testing, to learn more about proper patch testing.
This was first published in September 2005