Poor change control can send your organization's security tumbling. Follow these 5 steps for a strong change management program.
Mysterious activity shows up on key network devices at 2 a.m., setting off alarms in the network operations center. You're blocking Telnet and outside connections to the network, so this activity makes no sense. After 15 minutes of scrambling for answers, you find those connections weren't blocked at all. Earlier in the week someone made a change to grant network access to a vendor's technician, unintentionally creating a nightmare for you.
This simplistic example illustrates an important point: all too often, inadequate change control is the root cause of many headaches and sleepless nights for secu-rity managers. Keeping networks locked down is incredibly difficult because networks are in constant flux; patches must be applied, partners must be granted access and new services must be provisioned. One unmanaged change could send the neatly aligned and secured blocks of your network tumbling to the ground.
Change management can mean a number of things to security managers. To some, change management constitutes the processes in place to enact changes in the IT environment, such as new firewall rules or router access control list entries and system upgrades. This process includes approvals, testing and scheduling. Too frequently, though, IT security teams equate the concept of change management with configuration control and management, which includes activities like file integrity monitoring and system scanning. Although they are distinct concepts, both are vitally important to security, and the ties between change management and configuration management are getting stronger.
Security's challenge is to integrate itself into existing IT change management processes, while developing change and configuration management processes for systems and devices it controls. Organizations must include security to help evaluate requested and planned system changes and advise business management of associated risks. Getting a seat at the table is becoming easier because organizations understand the risks some IT changes could introduce.
However, the security team should act as a trusted adviser, not a dictator of whether changes are allowed or not, and demonstrate willingness to work with business owners and facilitate business initiatives.
Let's examine security's role in five key steps for a well-developed change management program.
1. REQUEST & APPROVAL
The initial phase of a change management process is the request for change, or RFC. In most cases, this will be entered into a change management application and submitted for consideration by the change advisory board (CAB). Usually this board includes representatives from the help desk, network operations, servers and system administration, security, internal audit, database management and application development. Other members may represent legal, operations and administrative departments.
Once the CAB allows the RFC to proceed, it must progress through a chain of approval that allows all stakeholders involved to determine whether the change should be executed. Information security teams should be involved in the approval process, and make approval decisions based on perceived risk to the organization for the requested change. This initial phase of the change management process should have a built-in feedback loop, so approval workflow members can ask the change requestor questions or discuss ramifications with other stakeholders.
- Ensure the request comes from a valid source that is authenticated and authorized to request a change by checking user accounts and repositories, identity management systems and log files.
- Establish an audit trail for change requests--who made the request, who logged in to the change management system to review the change, who was involved in approval, and who made comments on the change.
For security teams implementing changes for their own systems, such as IPSes and firewalls, important in-formation to provide in the RFC includes the actual technical changes to be made, systems to be affected, and the possible risks to those or other systems in the environment. Potential risks resulting from not implementing the change should also be recorded.
2. PLANNING & TESTING
The planning phase of change management is where real scrutiny of proposed changes occurs. During this phase, security teams should assess in detail the change's impacts to the organization's risk posture. For example, security may determine that granting network access to a partner would open critical resources to new threats, or that a software implementation is risky due to known vulnerabilities.
Implementation and back-out plans should be provided by the change requestor and any other groups involved in implementation. These should also be assessed in some detail by security personnel to ensure that organizational policies are adhered to and that the plans are reasonable. For security-related changes, the planning phase should also include a determination of personnel that will need to be involved in the change, and very specific criteria used to determine if back-out is necessary.
The testing phase in-volves lab testing and simulation of the proposed changes and back-out plans. Standard organizational policies and procedures regarding lab testing should be adhered to, and the results of testing should be clearly articulated to all members of the CAB. For changes involving other groups' systems and applications, security teams will often perform some form of risk assessment after standard functionality testing is complete. For example, they might perform a vulnerability scan against new systems or applications. Security teams testing their own changes and back-out plans for security devices and applications should also follow approved lab testing procedures.
3. SCHEDULING & COMMUNICATION
Scheduling changes is straightforward. All interested and affected parties should make sure the changes are scheduled to take place at a reasonable time. In many organizations, there are standard change windows--usually late night or early morning when fewer people are working and less business is being conducted--and these should be used whenever possible.
Effective communication is vital here. The team in charge of implementing the change should communicate by email and phone that the change is occurring and specify when it will occur. Who should be notified will differ from one organization to another, but an excellent rule of thumb is to contact everyone on the incident-response team's notification list, often referred to as a "call tree." Security teams should be notified of impending changes, as should IT operations teams and other groups that may be affected. In the case of security changes, security crews must contact everyone.
The key to successful change scheduling and notification is to turn it into a well-documented process with checklists. With some change management tools, this functionality is a standard component and can usually be accomplished within the application workflow, which in turn is linked to enterprise contact repositories.
During this phase, changes are implemented according to plan. Security team members will either have a standard on-call rotation where someone stands by, or they will be implementing the change if it is related to security systems or applications. Any changes deemed capable of causing significant security risks, or changes to mission-critical systems, should be monitored carefully. Security teams should also leverage monitoring solutions like IDS during changes when possible.
For changes that are problematic or need to be backed out, reaching staff members can be difficult during the standard off-hours change windows, which makes it vitally important that security personnel be available in some capacity
5. DOCUMENTATION & FOLLOW-UP
Documentation of all changes is critical for maintaining a sound audit trail. All configuration changes should be documented in a centralized database, and all approvals, comments, testing plans, back-out plans and other workflow-related documentation should be retained according to organizational policy. One reason for doing this is regulatory compliance (see "E-commerce Control," below). For any systems that store, process or transmit sensitive data that is covered under mandates like Sarbanes-Oxley and the PCI Data Security Standard, auditors will often ask to see documentation for all changes made since the last audit. Audit trails of changes also are needed for troubleshooting systems and applications, as well as security investigations and incident-handling procedures.
Follow-up meetings or analysis should occur whenever changes do not proceed as planned or a back-out is necessary. Any security issues caused by changes, such as data exposure or the introduction of system and network vulnerabilities, should be the basis for follow-up meetings led by security staff.
For security teams, there are a few points to keep in mind to avoid pitfalls with change management:
- Understand the distinction between change management and configuration management. Change management is a process framework, aided by technology. Configuration management is more often equated with tools themselves, like a software agent on a server that monitors critical files for changes or a configuration database (see "Tools for Change," below). However, well-designed configuration management can greatly facilitate proper change management for security teams.
- The biggest obstacles to achieving effective, risk-focused change management are people and processes, not technology. Focus on working with peers in different IT groups and business units to understand security's role in the change management workflow. "Security should become more of an 'active partner.' Tools should be less of a discussion than the policy and process creation," says Kris Brittain, a Gartner vice president.
- When you evaluate configuration monitoring and implementation tools, look for those that can be integrated into organization-wide change management products. Don't create silos for security and security products.
Although configuration monitoring tools are beginning to integrate with change management products, convergence of the two is a ways off. For now, security professionals should focus on policies and processes, refine internal processes for risk analysis of proposed changes, and ensure that proper audit trails are kept of all steps in the change management workflow.