The numbers tell you what you already know -- you're neck deep in a rising flood of new security vulnerabilities. Carnegie Mellon's Computer Emergency Response Team (CERT) reported 2,437 computer vulnerabilities in 2001, more than double the previous year's total. That's an average of about 10 every working day -- a lot to read about, much less deal with.
"Dealing with security vulnerabilities presents real problems," says Claxton Francis, director of information systems for the New York-based nonprofit Natural Resources Defense Council (NRDC). "I have to spend time each day reviewing the latest issues and evaluating the level of exposure to decide whether to deal with the problem immediately or wait for the next scheduled maintenance window."
It's not impossible to sift through this flood of information, find what's relevant to your organization and take swift action to prevent intrusions -- if you have a plan. The template is simple, but the devil is in the details of your environment. Broadly speaking, there are four steps to sound vulnerability management:
- Inventory your systems. Know exactly what you're running so you know exactly what to worry about.
- Manage the flow of information. Determine which information resources help you focus exclusively on the vulnerabilities that affect your systems.
- Assess the information. Evaluate the actual risk to your organization's systems security.
- Plan for response. Develop standard procedures to translate information into action.
Don't assume that large enterprises solve the problem simply by throwing people at it. No matter what the size of your organization, vulnerability management isn't simply a matter of resources -- it's about process. For example, one large international conglomerate (200,000-plus employees) created an 80-person staff dedicated to vulnerability management and patch deployment. Despite having dedicated labs in each business unit to test patches and fixes, the company still couldn't keep up.
"Prioritization," declares Bret Sigillo, a senior consultant and director of marketing for the managed services division of Predictive Systems, a security and network infrastructure consulting company hired by the conglomerate to find a way out of this quagmire. "They needed help in figuring out what problem needs their attention first."
Implementing sound vulnerability management practices costs time and money. But you need to weigh your costs against the potential consequences of intrusions. After all, companies lost millions last year to attacks that exploited known vulnerabilities. Code Red and Nimda cost companies worldwide an estimated $2 billion in damaged computing resources and downtime, according to Computer Economics. More than 600,000 servers were infected by Code Red, although the vulnerabilities it exploited had been published and a patch was available about a month before the worm was released. The patch for Nimda was available up to a year before it made its debut, but the worm still infected 160,000 hosts at its peak. Effective vulnerability management could have saved much of the cost of these and other viruses, worms and electronic exploits.
1. Inventory Your Systems
To get what you need, you have to know what you've got. Identifying deployed technologies yield the first cut at the security vulnerabilities that place your organization at risk. For many organizations, this alone is a major project, as their networks have grown without adequate documentation.
Here are the keys to taking stock of your systems:
Classify your network assets by platform. That's the first thing you'll look for when a new vulnerability is reported. Conduct and maintain a complete inventory of the hardware and software, including the versions of software and firmware and any patches or upgrades that have been installed. For example, if a vulnerability affecting only older versions of Sun Solaris is announced, you need to know which -- if any -- of your Solaris systems are vulnerable.
Determine risk potential. Identify the business exposure of each technology on your network. Think about the impact on your business if each of these elements was compromised or made unavailable. Which systems and software make up the critical core of your network?
Know what defensive tools you have in place. There are many kinds of defenses you can deploy, such as router filters, system logging and intrusion detection systems.
Many corporations use system and network vulnerability scanners to identify known bugs and holes in their infrastructure. Extensible yet easy-to-use software scanners are available from vendors such as Internet Security Systems, Foundstone, BindView, Symantec, Harris and others. These scanners probe for vulnerabilities using a database of already-catalogued system weaknesses. While this approach allows users to survey systems for hundreds of vulnerabilities, these products won't detect newly discovered holes until database signatures have been updated.
Regardless of whether you use a vulnerability scanner, maintaining an up-to-date inventory of your systems is the first critical step in knowing what to do when the next Big Vulnerability is publicized.
"The inventory process is still an issue for many organizations," says David Goldsmith, New York regional director of professional services for information security firm @stake. "The problem scales with the size of the organization -- if your network is small, manual systems may work well. But as the size of your network increases, collecting and maintaining the inventory data becomes a major undertaking, requiring automated tools.
Taking stock of large networks requires more than legwork and spreadsheets.
Several products automate collection of hardware and software information on workstations, giving sysadmins and infosec managers a quick reference to which systems may be at risk when vulnerabilities are announced. The various products are comprised of an agent that runs on the computers being inventoried and a database server to aggregate the information collected for reports. The products inventory hardware configurations -- including processor types and speeds, RAM, total and available disk space, network and video cards and other add-ons, as well as software information, such as OS version and applications (including patches and fixes).
Inventory tools have a close cousin in security configuration/patch management tools (see "Patching Across the Enterprise").
Inventory products include the following:
AssetMetrix (www.assetmetrix.com) is a service-based offering that uses e-mail to deploy its agents to PCs. AssetMetrix agents are manually run by users, with reports delivered via a Web-based interface. In addition to workstation information, AssetMetrix reports the configuration of PDAs connected to PCs.
Centennial Discovery (www.centennial.co.uk) provides an inventory agent that can be downloaded at login, by e-mail, from an intranet site or directly on a PC. Centennial supports Windows clients and Microsoft LAN Manager/NT/2000 and Novell Netware/ Intranetware network environments.
LANAuditor (www.lanauditor.com) is run from a central server, either manually or via a network login script. The LANAuditor agent can inventory an "average" PC's hardware in 15 seconds or conduct a full scan of hardware and software in about three minutes for a fully loaded system, according to the company. LANAuditor uses Microsoft Access database files and SQL queries to create its reports. It inventories Windows and Macintosh clients. LANAuditor offers a free 25-node evaluation version, which can be downloaded from its Web site.
Tally Systems (www.tallysystems.com) offers TS.Census, WebCensus, QuickCensus and PowerCensus, a comprehensive suite of inventory tools for Windows workstations ranging from Web-based subscription services to Microsoft SMS-compatible agents.
Tivoli Inventory (www.tivoli.com) supports Solaris, AIX, HP/UX, Red Hat and other *nix-based platforms, in addition to Windows workstations. Supports Oracle, Sybase, SQL Server and DB2 for data management and reporting.
2. Manage the Flow of Information
Finding security information on the 'Net isn't a problem -- a search of "computer security advisories" turned up about 62,000 pages in Google's Web index. But what about getting information that's relevant to your systems and software in a timely and practical manner without being overwhelmed? Now that's a problem.
Once you have an inventory, you can begin to filter vulnerability reports. If you're an all-Microsoft shop, Solaris and Linux vulnerabilities are only of passing interest. If your e-business is supported by an HTTP Web Server and WebSphere application server, alerts about IBM are a top priority. You have options for getting the information you need, with a catch -- you get what you pay for. Fee-based services provide a level of customized information and timeliness that free Internet services can't match. But if the cost is prohibitive, some free sites are excellent resources.
Three of the leaders in the security intelligence industry, SecurityFocus, Vigilinx and SecurityGlobal.net, offer strong fee-based services.
SecurityFocus' Security Intelligence Alert (SIA) service includes information on systems affected, a technical discussion and analysis of the problem and its potential impact, sample exploit code (if available) and mitigation strategies. Users decide the platforms and issues for which they will get alerts. While some of this information is available on the SecurityFocus free site, the SIA alerts include ratings of the severity, urgency, impact and credibility, as well as additional technical details.
SIA alerts are released to subscribers about two days before the abridged free versions are posted on the SecurityFocus Web site, so timeliness is a factor. Full SIA alerts are delivered by e-mail or fax. Subscribers can receive high-priority alerts by phone or SMS message.
Access to this enhanced information is priced at $5,900 per user per year for between 3 and 15 users.
Vigilinx's IntelliSHIELD also allows subscribers to create profiles detailing the technologies they use. Vigilinx analysts assign each alert a severity level, which is used to determine how customers are notified (e.g., e-mail, pager). In addition to technology-based alerts, Vigilinx offers "Intelligence Bulletins," which describe security challenges posed by new technologies as well as attack trends and techniques. IntelliSHIELD also issues "Geopolitical Analysis Reports," which examine world events and their potential repercussions in cyberspace.
The IntelliSHIELD Web-based interface also provides a central repository for information on your organization's response to vulnerabilities, offering reports on which systems have been patched and which are still vulnerable to attack. Maintaining the repository is manual, requiring users to enter in the details of fixes they have applied to systems.
IntelliSHIELD is priced on a per-user basis, starting at $40,000 per year for six users.
SecurityGlobal.net's SecurityTracker offers Web-based services for both end users and security services, such as managed security providers, consultants and integrators. Users can specify which systems they want to receive vulnerability notices about through an SSL-secured browser. Notices are sent via e-mail. Users can get all the information in the e-mail, or choose to receive basic information and get the full story by logging into the Web site.
SecurityGlobal.net also sends free weekly e-mail summaries of new vulnerabilities and maintains a free listing of vulnerabilities on its Web site. End-user pricing starts at $695 per month for two users. Service provider contracts are negotiated based on the service Security-Global.net provides. Free trial subscriptions are available.
While the fee-based services offer the most timely and targeted vulnerability information, not all organizations can afford them. With some effort, free services can be used for effective information management.
SecurityFocus and ICAT are among the leading free sources of security information. They can be valuable tools if you focus on searching for the information relevant to your systems.
SecurityFocus is a good place to look while drinking your morning coffee, but you may have to do some work to find out what you specifically need. The site offers a number of search tools to allow users to find security information for a particular vendor or product or by the vulnerability identifiers issued by BugTraq and the CVE Project, the two major catalogers of security problems. The strategy here is to search for vulnerabilities affecting the systems in your inventory.
The site's "Vulnerabilities" and "Advisories" sections bring together security information from all of the major vendors, security sites and researchers into an easy-to-use portal. The descriptions of vulnerabilities and solutions are brief, clear and easy to read. Links to exploits and patches are provided when available. While the alerts don't rate the relative severity of each problem, the discussion sections usually provide enough information for security pros to evaluate threats themselves.
The ICAT Metabase is the National Institute of Standards and Technology's central index to sites containing vulnerability and patch information. If SecurityFocus is the morning paper for vulnerability management, ICAT is an annotated reference book. ICAT provides a rich set of search functions, allowing users to zero in on the vulnerabilities relevant to their systems. While the ICAT site's help file states that the database is updated "at least monthly," many notices are posted within a day or so of their announcement.
ICAT's entries are less informative than those in the SecurityFocus database, but this is by design -- ICAT is meant to be an index, and each entry has pointers to sites where additional information can be found.
While both SecurityFocus and ICAT do a good job of integrating large amounts of information into searchable portals, you have to be proactive. Two other free services, Cassandra and Security Alert Consensus, send customized vulnerability information, although they lack the timeliness of paid services.
Cassandra, operated by Purdue University's Center for Education and Research in Information Assurance and Security (CERIAS), uses the ICAT database to provide customized e-mail notifications of vulnerabilities. Cassandra (named for the woman who warned the Trojans that bringing that cool Greek horse into the city might not be such a great idea) allows you to set up custom system profiles. You can further narrow the selection of vulnerabilities by adding keywords. For example, you might be interested in Perl vulnerabilities, but only if they mention "CGI" or "Web." When new vulnerabilities meet your criteria, you'll receive an e-mail with links to further information.
Timeliness can be an issue, as Cassandra relies on the ICAT database, which, as noted earlier, is sometimes updated only monthly. Cassandra may be better used as a backup source of information.
Security Alert Consensus, a service of the Systems Administration and Network Security (SANS) Institute, provides a weekly e-mail roundup of significant security warnings, customized to your interests. However, a week is a long time in the computer security world. You can also receive news announcements about both commercial and non-commercial security tools. SANS also offers a monthly roundup of Windows-specific security information (Windows Security Digest) and a weekly summary of security news (SANS News-Bites). Like Cassandra, SANS is a good backup source of information and provides valuable background material for security professionals.
Sharing Information: ISACs and InfraGard
All of the information sources discussed so far are "broadcasters" -- they aggregate information and present it to subscribers. While you can tailor some of these services to show a subset of the available information, the flow is one way. Many organizations are finding value in sharing information about problems, vulnerabilities and incidents with their peers through the Information Sharing and Analysis Centers (ISACs). Each ISAC brings together a community of organizations with similar interests:
- Financial Services
- Information Technology
- Water Industry
There's also a Worldwide ISAC, which is open to all companies. ISAC members exchange security information anonymously, which promotes sharing without exposure, according to Suzanne Gorman, treasurer of the Financial Services ISAC.
"Think about it this way: A bank or brokerage house experiencing a major attack would want to know if others in their industry were also being targeted; an attack against the U.S. financial sector demands different responses than an attack on a single organization," Gorman says. "However, there's a catch -- no bank or brokerage wants to let the world know about their security problems. The ISAC provides a forum for safely and anonymously sharing information among a community of organizations who are competitors in other business situations."
The cost to join an ISAC varies. Most charge annual fees of between $5,000 and $10,000 to fund their activities.
Predictive Systems, which runs a number of the industry ISACs, offers a "Corporate ISAC" program to allow larger organizations to build internal information sharing and analysis centers. The Corporate ISAC program includes data feeds from Predictive's @lertnet network of industry, law enforcement and government information sources. A Web portal allows distribution of security information and tracking of fixes. Predictive's data feeds include risk ratings for each reported vulnerability, and their analysts provide additional background materials and weed out hoaxes. Pricing for the Corporate ISAC program depends on the options chosen.
The FBI's InfraGard program offers businesses a forum for sharing information on cyberthreats and security solutions. InfraGard is run out of the FBI's 56 field offices via local chapters. As with the ISACs, InfraGard members report incidents to a central clearinghouse, which processes and distributes information.
"The purpose of InfraGard is to get private sector companies talking to each other about physical and cyberthreats," says Thomas J. Van Nuys, supervisory special agent in charge of domestic terrorism for the FBI's Chicago field office. "The FBI provides resources and information, such as alerts from the National Information Protection Center (NIPC) and other sources. We're trying to get rid of the misconception that the FBI sits on information."
To join InfraGard, apply at a local FBI field office. Applicants are subject to a background check. "We don't want to pass InfraGard information to terrorists or criminals," Van Nuys says.
No matter which sources of information your organization uses, the key to making the most of them is consistent monitoring. Checking the "threat radar" for potential problems should be just as much of a part of your daily routine as checking system logs and making backups. In larger organizations, this monitoring task may be assigned to an information security department. Smaller companies may not have a full-time, dedicated security staff.
In this case, IT management will need to decide who's responsible for monitoring security information, how it will be distributed and how it will be evaluated.
3. Assess the Information
Alerts -- even alerts about vulnerabilities in your systems -- don't tell you everything you need to know. Assessing the level of the threat to your organization -- hence the level of response -- must be an integral part of your vulnerability management strategy.
Say you've spotted a potential problem while searching SecurityFocus -- a new, remotely exploitable vulnerability that could provide an attacker with root-level access to the operating system of your Apache Web server. Before you start notifying systems administrators and downloading patches, you have some analysis to do.
The answers to the following questions determine if you should ignore a given vulnerability, put it on the "To Do" list or drop everything and ring all hands on deck:
- Does the problem affect a technology that's in use in your organization? Having an accurate inventory of your systems and software is crucial to answering this key question. If the answer is no, you can resume your daily routine.
- If the vulnerable technology is in use on your network, are you running the problem version (or component)? In many cases, vulnerabilities are specific to certain revisions of software or affect optionally enabled components. You may be running a version -- older or newer -- that isn't vulnerable. Or, it may affect a version that you haven't deployed but are planning to roll out.
- What business resources are at risk? If a vulnerability is relevant to your network, determining which systems are potential targets is vital. If "core" systems such as your firewall, DNS, mail or key business servers could be affected, the risk is more severe than if a seldom-used FTP server is a potential target. Your risk assessment should also consider the location of the affected server. For example, is it a bastion host in the DMZ or a database in the private network?
- Can the vulnerability be exploited remotely? A security problem that can be exploited by an anonymous attacker over the Internet is much more of a threat for most organizations than one that requires physical access to a system console.
- What's the potential result of a successful attack? Some vulnerabilities allow an attacker to access, modify or delete confidential information. Others can crash a server or use it to stage an attack on third parties. Still others reveal non-critical data. You need to consider the applications running on the vulnerable server, the network resources available to it and the data stored on it. Having customers' credit card information at risk of disclosure is probably more serious than having your Web site DoS'd.
- How common is the platform being threatened? A vulnerability in Windows 2000 or the IIS Web server is much more likely to draw the attention of attackers than a flaw in, say, an ERP application. While security through obscurity isn't a solution, it may buy your organization time to implement fixes.
- Are exploit tools and scripts available for the vulnerability? Let's face it -- many so-called crackers are nothing more than script-kiddies looking for point-and-click attacks. On the other hand, if a vulnerability requires a high skill level to exploit, the universe of potential attackers is smaller.
- What steps can you take/have you taken to mitigate the risk? Some vulnerability announcements are accompanied by patches. Others include instructions for bolstering defenses until the vendor releases a fix or work-around.
A buffer overflow flaw in the Solaris and AIX server operating systems, which was announced last December, illustrates several of the points these questions raise:
- The vulnerability affected all versions of Solaris and the current AIX release, as well as an earlier release, potentially impacting many systems.
- An attacker could gain super-user or root-level privileges on servers through the login program.
- The vulnerability could be exploited remotely through terminal connection programs, such as Telnet and rlogin.
- An exploit was already public at the time of the announcement.
- The urgency prompted CERT and Internet Security Systems to recommend disabling default terminal connection services and installing Secure Shell (SSH)until a patch became available.
4. Plan for Response
Your phone is ringing. It's The Boss. A new vulnerability has been reported on CNN. Systems all over the Internet are being scanned for the problem by eager script-kiddies, and a number of Web sites have been defaced. "What are we doing about this?" she asks.
It will be a lot easier to handle the boss' call if you are practicing good vulnerability management -- chances are, you've already dealt with the problem or determined it has no impact on your network. That kind of peace of mind depends on implementing the last part of your vulnerability management process: the action plan for response. The template for this plan should include the following:
- A brief description of the problem/vulnerability.
- An inventory of the systems affected by the vulnerability.
- A description of the business systems and processes threatened by the problem.
- Contact names and numbers for the administrators and line business managers of affected systems.
- The ultimate fix for the problem (e.g., a patch or major configuration change).
- The transitional fix for the problem (e.g., temporarily turning off the vulnerable service).
- The plan for implementing the temporary fix -- with a rollback plan in case of problems.
- The plan for testing the permanent fix.
- The plan for implementing the permanent fix -- with a rollback plan in case of problems.
Prioritizing use of organization resources is crucial, says @stake's Goldsmith. "You need to look at your business needs and the characteristics of the vulnerability. If you are dealing with a remotely exploitable bug, you might want to schedule your business-critical Internet-facing systems for patching first to reduce the overall risk level as quickly as possible."
Different courses of action are possible in each case, depending on the available options, urgency of the problem and available resources:
Apply a patch. Most vendor-supplied vulnerability alerts will include software patches or upgrades to eliminate the security problems. It may be tempting to just apply the patch, but hasty action can lead to further headaches. Vendors are usually under a lot of pressure to get security patches out as quickly as possible and may not have fully tested them. A buggy patch could convert a potential threat into a real problem. Microsoft, for example, had to replace several flawed patches last year -- twice in one case, because the replacement patch was also flawed.
Patching presents system administrators and business managers with a number of thorny issues. The problem system may become more vulnerable while it's being worked on. Many patches require systems to be rebooted -- and managers may be reluctant to make their key systems unavailable during the business day. Deploying a patch on a large number of systems -- say, all of your desktops -- may be disruptive. Plus, the presence of patched and unpatched systems on the same network may lead to incompatibility issues. Several vendor tools automate the deployment of patches and fixes to address this potential problem.
In the most extreme cases, patches to fix one security issue may cause another, more serious (and undocumented) vulnerability. "Sometimes you can kill the patient with the medicine," says the NRDC's Francis.
The two keys to successfully deploying patches are (1) testing the new software in an environment similar to your production networks before rolling it out, and (2) getting cooperation from the line business managers whose departments are affected by the update. For many organizations, predeployment testing is a particular problem, since it requires additional time, people and technology. As systems grow more complex, developing plans that test all of an application's functions becomes more difficult and time-consuming. Many organizations have chosen to compromise, using limited deployment of patches for serious issues in production.
While most security vulnerabilities will be remedied with a patch or software update, there are other actions you can take to reduce your risk until new software can be properly tested and deployed:
Turn off vulnerable services at the host level if they aren't needed.
Hopefully, you hardened your systems a long time ago, perhaps when you installed them. However, unneeded services can be overlooked. For example, the infamous IIS Remote Data Service vulnerability found in 1998 uses a feature providing access to SQL databases from Microsoft's IIS Web server to execute privileged commands. Many Webmasters who didn't need to access SQL data could have avoided defacements of their sites simply by turning this feature off.1
Adjust system configurations to remove the vulnerability. You may be able to mitigate risk by changing the way a vulnerable service is configured. For example, a recently reported vulnerability in Lotus' Domino Server allowed nonprivileged users to access a file called the Web Administrator Template and gain super-user access to the server. The problem could be easily fixed by changing the permissions on the offending file or removing it from the publicly accessible part of the server.
Adjust firewall rules to prevent access to vulnerable systems or services. Sometimes, unpatched systems can be protected by blocking access to the vulnerable service at the perimeter of your network until fixes can be made. Of course, this won't repel the inside attacker, and shutting down external access to vulnerable services may not be practical -- an SMTP server becomes pretty useless if it can't accept mail from the outside world.
Use other perimeter defenses. The last few major Internet-wide security events (Nimda, Goner) used e-mail as a significant transport vector. Many organizations filter e-mail attachments with suspicious file types (like .scr or .vbs). Another approach to this problem is to drop messages with content that matches known infected messages. For example, the message carrying the Goner virus had a number of signatures to key on -- the file attachment named "Gone.scr" and distinctive phrases, such as "I'm in a hurry."
As networks get more complex, with new and upgraded software adding functionality, each day will bring new vulnerabilities to check out. Many of these vulnerabilities will spawn potentially dangerous exploits, making a coherent, consistent vulnerability management strategy an even more vital part of your security toolkit.
About the author:
Al Berg, CISSP, is a contributing editor for Information Security and a technical director in the corporate information security department of a firm providing data processing services to the financial industry. He has written numerous articles on information security and lectured worldwide on infosec topics.