George Wrenn, CISSP, ISSEP
Published: 15 Nov 2004
Time is the enemy of every security manager charged with patching.
Despite (or perhaps because of) vendors' attempts to release patches on regular intervals, enterprises are still racing to seal holes in their infrastructures. Every extra minute a system remains unpatched is another opportunity for worms, backdoors, rooters and Trojans to infiltrate the network.
The clock starts the day the vulnerability is announced and, in many organizations, never stops. We all know the basics of how to roll out patches across the enterprise, so why is success so elusive?
In many enterprises, patching is a messy, time-consuming process through which security teams must lab test new code before welding it onto production machines. The most common mistake is repeating the arduous process with each deployment rather than building a process that makes deployments successively easier.
Think Six Sigma: Every time you roll out a patch, look for obstacles and make adjustments that simplify the patch process for the current and subsequent cycles. Through continuous process improvement, you can better plan patch deployments; save time and money; reduce errors and disruptions; and improve overall security.
Know thy network
If your cycle time is a month and vulnerabilities are being announced at the rate of two per week, the math is working against you. By the time you make it halfway through your first cycle, there are four new vulnerabilities that need remediation.
Simply increasing the speed of patch deployments isn't the remedy. Effective and expeditious patch management is as much about planning as it is implementation. This is where asset management-knowing what's on the network-comes into play. Inventorying and mapping your assets give you the ability to prioritize patches and their installations. Target high priority and high-value assets, then stage the remaining systems sequentially.
But networks are dynamic environments; inventorying and asset management are continuous processes. Each time you do them, you learn more about your network and can adjust patching plans accordingly.
Inventorying assets involves various security tools, including vulnerability scanners such as Nessus, Internet Security Systems' Internet Scanner and eEye Digital Security's Retina; mapping tools such as Nmap and p0f; and patch-level detection software. Some vendors, such as Tenable Network Security, are adding OS fingerprinting to their discovery tools, allowing for the detection of devices and information about what OSes and services they're running. Many automated patching systems include network mapping engines and service/ OS discovery tools that list machine names, IP addresses, OS version, risk level, vulnerabilities and missing patches.
Scanning and discovery tools have different capabilities and degrees of accuracy-none is 100% reliable. False positives and false negatives, from misidentifying OSes to failing to identify vulnerabilities, are common among all devices. Best practice is to use multiple discovery tools to check and verify the results of other scans.
Your network scans will be more accurate and efficient in identifying vulnerable systems as your asset management becomes more comprehensive. This, in turn, will enable you to develop sound policies backed by management and strong operational agreements with business and IT support units. These will translate into quicker, less disruptive system scans and prioritized patch deployment.
Knowledge is a change catalyst
The benefits of network discovery and asset management processes are multifold. Not only do they help you better plan patch rollouts, but they empower you to craft policies and architectural changes to improve subsequent patch deployments and overall network security.
Consider this: Devices that are intermittently connected to the network-roving laptops, partner and contractor servers, etc.-account for most of the systems missed by network scans and consequent patch deployments. Bringing them into the fold will dramatically improve your coverage and will enable the deployment of endpoint security products to check devices against security policies before allowing network connections.
An accurate inventory process will identify machines, OSes and services and give you the opportunity to standardize platforms and images. Reducing the number of accepted OS versions will simplify future patch rollouts since you will only have to test against future platforms. Enforcing standard images will reduce the chance that a patch will blow up a particular machine because of unauthorized apps or utilities.
When you have a better sense of what's on your network, you can rearchitect to both improve security and smooth out future patch deployments. For instance, after discovering that your financial databases are distributed across your network, you could use the asset management information to move them into their own subnet and protect them with a DMZ and IPS. That way, when a patch is released for those systems, you know where they are and that they need to be patched first.
With the way today's networks are put together, it's no wonder that worms spread like wildfire. Networks are often hastily pieced together with a business model, not security, in mind, leaving them wide open to attack. The defense-in-depth scheme will better secure your enterprise in the long run-and slow the vulnerability exposure clock.
Managing people and patches
It only takes one patch deployment cycle to discover which departments cooperate and which stymie the process.
Welding shut security holes is as much about managing people as it is the patches. You need to develop lines of communications, cooperation agreements and processes for patch deployment. To security managers, it's a matter of utmost urgency, but to business managers -- those who own different pieces of the infrastructure -- it means lost production time as their servers are taken offline for reboots.
You need to build and formalize a working relationship by stressing to each department the importance of patching.
Security departments should set expectations and guidelines for patching across the enterprise. With historical data and a clear sense of ROI and the criticality of patching, you can create a system in which business units are required to install critical patches immediately; moderate patches during predefined maintenance cycles; and routine patches during periodic updates. Enlist business units to carry the burden of patching and make them accountable, enforcing requirements and maintaining standards through SLAs.
You can use the information gleaned from successful, as well as botched, patch deployments to show each uncooperative department how its delay in deploying a critical Windows patch lead to a worm infection that cost significantly more downtime and remediation than the patch would have cost. Because time and resources mean money to business managers, show them the cost of failure.
This will keep patching from disrupting critical, revenue-producing processes while engaging cooperation from business units to address immediate threats.
Deploying patches is only half of the patch management process, the other half is verification. Whether you're using patch management systems, automated tools like Windows Update or SUS, or manually installing patches, you always need to verify that the patch is installed correctly and is working properly.
Automated patch management tools are infamous for updating the registry keys to reflect full installation, yet failing to recognize that the patch download was disrupted-thus, incorrectly showing that the patch was successfully deployed. Then there are the patches that just don't work or undo the fixes installed by previous patches. If these issues aren't corrected, they will create problems that will slow future patching and performance.
This is especially important if the security department isn't the one doing the patching. Network and business unit managers will swear up and down that they deployed a patch, but verification is up to you.
You'll need to pull out those same discovery, vulnerability assessment and penetration testing tools we talked about earlier. When used in concert, these tools will verify the effectiveness of your patch rollout, identify which systems need further work and measure the level of residual exposure.
By taking the time to verify patches, you're essentially saving time during next rollout cycle. You won't be troubled by the mistakes of previous deployments and will be able to focus on the work in front of you.
Each system that you fail to remediate in a given cycle is a threat to your enterprise. It may be a home PC connecting through a VPN, or a developer's server attached to a production environment. The point of the patch management process is to get as close to 100% as possible.
What is the standard for patching? What percentage of systems should be patched within a week of a patch's release? What percentage of unpatched machines is an acceptable risk?
There are no right answers. Enterprises must develop their own metrics for measuring the success of their patch management programs. In the end, by employing these methods for continual improvement, your patch management system will reduce the vulnerability exposure and remediation time associated with enterprise-wide patch deployment.
About the Author:
George Wrenn, CISSP, ISSEP, is a technical editor for Information Security magazine and a security director at a financial services firm. He's also a graduate fellow at the Massachusetts Institute of Technology.