Published: 01 Nov 2003
We've barely recovered from the worm attacks led by Blaster this summer. Yet, by the time you read this, we may well be in the midst of the next malware disaster. Chances are we're no better prepared for the next assault, because our networks aren't properly partitioned and our operating systems are too frail to exist on a network connected to the Internet.
Vendors say their widgets will protect us from the next attack. Former cybersecurity czar Richard Clarke calls this a "widget mentality" -- the assumption that a defense-in-depth with overlapping firewalls, IDSes, IPSes and content filters will somehow save us. It won't.
Despite these cool technologies, our systems are still exploitable. Malware enters our networks through RAS modem pools, VPN tunnels, road-warrior laptops, open network shares, partner extranets and wireless access points. No single product or combination of security solutions can completely stop it.
Partitioning our trusted network into security zones is one approach that enables us to prevent a worm outbreak from spreading through the entire enterprise. Witness the Blaster attack, which hit the U.S. Navy hard, but was contained so that the rest of the Department of Defense wasn't affected. But containment isn't the same thing as prevention.
The problem with layering conventional point solutions is similar to the mishaps of Inspector Gadget, who had a number of wonderful tools at his command but could rarely capitalize on them. The "Inspector Gadget" approach to infosecurity fails because of "change creep," which leaves a mishmash of configurations and renders our poorly patched systems almost unmanageable.
Instead, enterprises need to make a top-down commitment to strict change control management. This involves a sea change in how your organization approaches security. This emphasis is on hardened, standardized systems -- with up-to-date patches and service packs -- that make your infrastructure more resistant to attack.
Change Control Is Good
Because your operating systems can't defend themselves, you need to make a simple decision: Do you want to manage your systems, or do you want the Internet to manage them for you? Designing a network that can be rapidly partitioned while it's under attack is great, but you could still lose an entire department to an infection. A better approach is to manage the risk on every single system.
System-based risk management lowers operational costs and increases security. It focuses on centrally managed change control. It requires a culture that never troubleshoots anything for any reason. If a machine acts funny, even a little bit, burn it to the ground and rebuild it from a known, good standard.
But what is change control management? In a nutshell, it's a series of policies, procedures and technologies that enable enterprises to effectively monitor, evaluate, direct and audit changes on multiple platforms and devices. Solid change control management will free you from widget mentality.
Policy Starts at the Top
Change control management is a political issue, not a technical problem. Your organization should have the same zero-tolerance policy towards unauthorized change as it has for weapons and drugs in the workplace. Senior management must fully understand and support this philosophy, and your operations team leaders must "buy in" if you want to protect your organization's assets and intellectual property.
The trick is to write, implement and enforce a policy that doesn't tolerate unauthorized change. A good example of change control management is the Securities Industry Automation Corp., the company that runs the New York (NYSE) and American (AMEX) stock exchanges. SIAC has high uptime records, which is necessary since every second of downtime costs serious money. What SIAC's secret? Shift leaders end each day by running the file integrity tool Tripwire, which shows if servers' states have changed. The shift doesn't go home until they resolve all unauthorized changes.
Effective change control policy begins in the data centers. The key is to set a minimum number of architectures with a goal of having a single enterprise-wide architecture based on a gold standard. Otherwise, you'll become a snowflake shop, where every server is unique -- just like a snowflake. Standardization makes rollback feasible and simple.
Your toughest battleground, of course, is the desktops -- moving from the notion that we have a right to load any software we want to on our work computers to one in which only authorized apps are allowed. This is a matter of reinforcing the axiom that risk accepted by one can affect all. Again, this is a battle of policy and perception, not technology.
Starting the Process
OK, you've committed to change control management as a matter of policy. In practice, how does an IT shop get from here to there-and stay there? Rather than create a new model, let's apply the ISO 17799 methodology: Plan, Do, Check, Act.
Plan. Start by establishing a baseline. How many servers can be created reliably from an existing build? How much time do you spend troubleshooting? Most organizations have no idea until they're required to document unplanned work. It's an indicator of the efficiency and effectiveness of your IT shop -- the less troubleshooting, the better.
It only takes a week or two to develop the baseline metrics you need to determine how far your organization is on -- or off -- the mark. These metrics are useful in securing senior management support, since they are tangible evidence of change. If you have a configuration control board, it's pretty easy to prove the changes were unauthorized.
Consider the VP of operations who thought his organization practiced strong change control management. Within a week of engaging a scanning service for his Internet-facing systems, a change was detected -- a new open port on his Web server farm. Yet, the change control board had no report of it -- clear evidence of poor change control and operational practices. With the scan report in hand, he had hard evidence the organization had a problem and the information needed to get it back on track.
Do. If you are a snowflake shop, start slowly. Follow the physician's credo, "First, do no harm." You're going to hit snags no matter what you do, but if you try to solve everything at once, the perception of failure may derail the whole program.
Look for a quick win and target the frailest computers first. Build a model operating system platform (fully patched) as your gold standard, and move those first systems' applications to it. Keep good records -- you'll learn a lot about your organization's ability to engineer new platforms as opposed to putting Band-Aids on old ones. Then put the rest of your enterprise on the gold standard, with priority on mission-critical systems.
Success is an ongoing process, and it's easy to slip. How can you implement repeatable builds, for example, if Microsoft continues to release patches every week? Who would have guessed that the lifespan of a gold build might only be three or four days? For most operating systems and applications, a patch is a matter of when, not if. Security and IT managers often say they don't have time to test and deploy patches, but they always find time to rebuild systems after they've been infected. When a worm like SQL Slammer comes along, their admins are suddenly patching like crazy with very little testing. Emergency patching is unplanned work. Patching without testing eventually results in snowflake servers, which lead to even more unplanned work.
This is precisely why you must strive to operate with the minimum possible number of architectures. A patch applied across the board to standardized systems merely changes one architecture. Just make sure you keep the image for that system up to date.
Check. Establish a regular program of using file integrity tools, scanners and other technologies to monitor your systems for change. The tools are readily available (see below).
What happens if your checks reveal that you are slipping, and that unplanned changes are happening? It's unavoidable; change happens. You may lose a few hours, but your checks will detect the flaw in your operation and you will correct it before it gets worse and spreads to other systems.
Your auditors can be strong allies in maintaining change control. We're often blind to our own errors, but trained auditors who understand the established controls can help you remain committed to a goal of zero unplanned changes.
Expect your change control management program to fail multiple times. The important thing is that you get it restarted. Change control management gets easier each time you restart the program.
Act. The goal of your follow-up is quality improvement. There will be days when it seems like you're barely holding your own. Yet, you will improve each time you discover a patch that wasn't applied to a system or applied prematurely, or find a sysadmin who made some adjustment in the operating system, but didn't follow proper procedure. Your corrective action strengthens your security level and change control program.
The core technologies to implement change control are change-detection tools like Microsoft's Group Policy, AIDE and Tripwire. Group Policy is a feature of Windows 2000/XP, which allows you to establish a policy or configuration and push it to all servers and desktops. AIDE is free and available for contemporary Unix systems, which is similar to the open-source version of Tripwire. And Tripwire, available for Windows and Unix, fingerprints a system the first time it runs, and then flags changes when it's run again.
Network vulnerability assessment scanners are also invaluable for detecting change. A number of commercial scanners and services are available, such as Qualys' QualysGuard, Foundstone's FS1000 Appliance and service, SAINT, eEye Digital Security's Retina and Internet Security Systems' Internet Scanner and Systems Scanner, as well as freeware classics Nmap and Nessus.
A word of caution: Network scanners sometimes break things. Tools like Nmap that simply probe ports are less likely to cause trouble than invasive scanners. The first few times you use any of these tools, run them on a subset of your network during a scheduled maintenance session or off-peak hours. There are also passive scanning tools like freeware p0f for OS fingerprinting and commercial-grade Tenable Network Security's NeVO for vulnerability detection, whose OS fingerprinting is P0F-based.
Change creeps into your network in many ways. Here are some solutions to help you bring it under control:
Patch management. If managing the patching process is getting the better of your organization, you are in good company. Tools like PatchLink, BigFix's BigFix Patch Manager and Citadel's Hercules can discover network vulnerabilities through scanners or agents, track changes in patching status and automate the patching process. After surviving two DCOM patches within days, patch automation is starting to sound good.
Remote clients. Outside connections -- from home offices, road warriors and partners -- are a real threat to change control. It's hard to know the state of a system outside of your network. Investigate solutions like Sygate's Secure Enterprise, which enforce patch levels, AV signatures and system configuration policies before allowing an endpoint to connect into your network.
Help desk calls. Complaints, problems and requests for new features are another significant driver for change, which must be managed. Well-funded organizations can leverage trouble ticket systems like Remedy to process and track changes. If your budget is tight, freeware Request Tracker is a great ticket tracking alternative. SANS has used Request Tracker to track questions and issues related to the GIAC certification for more than a year.
Perimeter security tools. IDSes, firewalls and other security widgets often have configuration files that require change management. This can be as simple as using a version control system such as CVS for Unix. This way you can check in a rule set and check it out to modify it. Microsoft shops can use commercial version control tools like Visual Studio. For the truly brave, there are Windows versions of CVS and RCS, but these implementations of classic Unix tools don't work perfectly in the Microsoft world.
Holding the Line
If you have effective change control management and a minimal number of architectures, you can use widgets to your advantage until software vendors -- especially Microsoft -- finally begin shipping secure, properly configured operating systems. It's much easier to tune IDSes if you're only protecting a single architecture. It's much easier to configure a firewall if you don't allow unnecessary services.
Widgets do have value, but they simply aren't enough. If your CEO is willing to make a commitment to implement hard-nosed, no exceptions, no excuses, change control management, you'll stem rising costs, increase your organization's efficiency and build tighter company-wide security. It takes discipline and persistence, but the end result is worth the effort.
About the author:
Stephen Northcutt is director of training and certification at the SANS Institute. He is a coauthor of several infosecurity books, including Network Intrusion Detection (Pearson Education, 2002) and Inside Perimeter Security (Que, 2002).