5 Steps for Developing Strong Change Management Program Best Practices

Poor change control and configuration management can affect the security of your systems and networks. Follow these five steps for a strong change management program.

This article can also be found in the Premium Editorial Download: Information Security magazine: Seven questions to ask before committing to SaaS:

Poor change control can send your organization's security tumbling. Follow these 5 steps for a strong change management program.

Mysterious activity shows up on key network devices at 2 a.m., setting off alarms in the network operations center. You're blocking Telnet and outside connections to the network, so this activity makes no sense. After 15 minutes of scrambling for answers, you find those connections weren't blocked at all. Earlier in the week someone made a change to grant network access to a vendor's technician, unintentionally creating a nightmare for you.

This simplistic example illustrates an important point: all too often, inadequate change control is the root cause of many headaches and sleepless nights for secu-rity managers. Keeping networks locked down is incredibly difficult because networks are in constant flux; patches must be applied, partners must be granted access and new services must be provisioned. One unmanaged change could send the neatly aligned and secured blocks of your network tumbling to the ground.

Change management can mean a number of things to security managers. To some, change management constitutes the processes in place to enact changes in the IT environment, such as new firewall rules or router access control list entries and system upgrades. This process includes approvals, testing and scheduling. Too frequently, though, IT security teams equate the concept of change management with configuration control and management, which includes activities like file integrity monitoring and system scanning. Although they are distinct concepts, both are vitally important to security, and the ties between change management and configuration management are getting stronger.

Security's challenge is to integrate itself into existing IT change management processes, while developing change and configuration management processes for systems and devices it controls. Organizations must include security to help evaluate requested and planned system changes and advise business management of associated risks. Getting a seat at the table is becoming easier because organizations understand the risks some IT changes could introduce.

However, the security team should act as a trusted adviser, not a dictator of whether changes are allowed or not, and demonstrate willingness to work with business owners and facilitate business initiatives.

Let's examine security's role in five key steps for a well-developed change management program.

The initial phase
of a change management process is the request for change, or RFC. In most cases, this will be entered into a change management application and submitted for consideration by the change advisory board (CAB). Usually this board includes representatives from the help desk, network operations, servers and system administration, security, internal audit, database management and application development. Other members may represent legal, operations and administrative departments.

Once the CAB allows the RFC to proceed, it must progress through a chain of approval that allows all stakeholders involved to determine whether the change should be executed. Information security teams should be involved in the approval process, and make approval decisions based on perceived risk to the organization for the requested change. This initial phase of the change management process should have a built-in feedback loop, so approval workflow members can ask the change requestor questions or discuss ramifications with other stakeholders.

Security must:

  • Ensure the request comes from a valid source that is authenticated and authorized to request a change by checking user accounts and repositories, identity management systems and log files.
  • Establish an audit trail for change requests--who made the request, who logged in to the change management system to review the change, who was involved in approval, and who made comments on the change.

For security teams implementing changes for their own systems, such as IPSes and firewalls, important in-formation to provide in the RFC includes the actual technical changes to be made, systems to be affected, and the possible risks to those or other systems in the environment. Potential risks resulting from not implementing the change should also be recorded. 


 Security's Role...

  1. Authenticate change requestor
  2. Establish an audit trail for requests


The planning phase of change management
is where real scrutiny of proposed changes occurs. During this phase, security teams should assess in detail the change's impacts to the organization's risk posture. For example, security may determine that granting network access to a partner would open critical resources to new threats, or that a software implementation is risky due to known vulnerabilities.

Implementation and back-out plans should be provided by the change requestor and any other groups involved in implementation. These should also be assessed in some detail by security personnel to ensure that organizational policies are adhered to and that the plans are reasonable. For security-related changes, the planning phase should also include a determination of personnel that will need to be involved in the change, and very specific criteria used to determine if back-out is necessary.

The testing phase in-volves lab testing and simulation of the proposed changes and back-out plans. Standard organizational policies and procedures regarding lab testing should be adhered to, and the results of testing should be clearly articulated to all members of the CAB. For changes involving other groups' systems and applications, security teams will often perform some form of risk assessment after standard functionality testing is complete. For example, they might perform a vulnerability scan against new systems or applications. Security teams testing their own changes and back-out plans for security devices and applications should also follow approved lab testing procedures.


Security's Role...

  1. Examine risk associated with changes
  2. Ensure back-out plans are within policy
  3. Perform risk assessments on changes in a lab setting


Planning & Implementation
Developing a change management program from the ground up is tough, but organizations can take steps for a smooth implementation.

Companies developing a change management program from the ground up often face daunting challenges. Effective change management relies on a reasonably accurate inventory of systems and applications, and greatly benefits from a data classification scheme for asset and data prioritization. Building system inventories and applying classification to assets and data are complex and often tedious tasks. And since change management is a discipline that encompasses the entire IT organization, political battles are common. These steps can help facilitate planning and implementations:

  • LEVERAGE your data classification scheme or other asset management initiatives to prioritize assets in terms of criticality and potential risk. Any stores of sensitive data or mission-critical systems that could affect business availability or integrity should be given high priority.
  • DEVELOP a master database of assets and system configurations that can be used for tracking. An example would be the Configuration Management Database (CMDB) used in ITIL configuration management implementations.
  • IMPLEMENT change management tools that allow simple integration with help-desk ticketing, user identity repositories, identity management products and asset management products.
  • DEFINE what changes, or types of changes, need to be managed versus those that do not. For example, adding a new user account to Active Directory may not require a full-blown change management process, but adding a rule to the firewall would.
  • DETERMINE the business owners and stakeholders who will participate in the process, and create a change advisory board.
  • CONSIDER compliance. If you are beholden to one or more regulations, ensure that impact to existing controls and documentation is an integral consideration for all participants in the change management workflow.

Several frameworks offer guidance for change and configuration management. The Information Technology Infrastructure Library (ITIL) includes detailed guidance for managing IT operations and infrastructure. Although ITIL has been criticized for being overly complex and difficult to implement, some experts recommend starting an ITIL implementation with its change management component as a foundation.



Scheduling changes is
straightforward. All interested and affected parties should make sure the changes are scheduled to take place at a reasonable time. In many organizations, there are standard change windows--usually late night or early morning when fewer people are working and less business is­ being conducted--and these should be used whenever possible.

Effective communication is vital here. The team in charge of implementing the change should communicate by email and phone that the change is occurring and specify when it will occur. Who should be notified will differ from one organization to another, but an excellent rule of thumb is to contact everyone on the incident-response team's notification list, often referred to as a "call tree." Security teams should be notified of impending changes, as should IT operations teams and other groups that may be affected. In the case of security changes, security crews must contact everyone.

The key to successful change scheduling and notification is to turn it into a well-documented process with checklists. With some change management tools, this functionality is a standard component and can usually be accomplished within the application workflow, which in turn is linked to enterprise contact repositories.


Security's Role...

  1. Ensure you know when changes are made
  2. Communicate security changes


During this phase,
changes are implemented according to plan. Security team members will either have a standard on-call rotation where someone stands by, or they will be implementing the change if it is related to security systems or applications. Any changes deemed capable of causing significant security risks, or changes to mission-critical systems, should be monitored carefully. Security teams should also leverage monitoring solutions like IDS during changes when possible.

For changes that are problematic or need to be backed out, reaching staff members can be difficult during the standard off-hours change windows, which makes it vitally important that security personnel be available in some capacity


Security's Role...

  1. Be on call, especially with problematic changes
  2. Monitor changes that could add risk to systems


Documentation of all
changes is critical for maintaining a sound audit trail. All configuration changes should be documented in a centralized database, and all approvals, comments, testing plans, back-out plans and other workflow-related documentation should be retained according to organizational policy. One reason for doing this is regulatory compliance (see "E-commerce Control," below). For any systems that store, process or transmit sensitive data that is covered under mandates like Sarbanes-Oxley and the PCI Data Security Standard, auditors will often ask to see documentation for all changes made since the last audit. Audit trails of changes also are needed for troubleshooting systems and applications, as well as security investigations and incident-handling procedures.

Follow-up meetings or analysis should occur whenever changes do not proceed as planned or a back-out is necessary. Any security issues caused by changes, such as data exposure or the introduction of system and network vulnerabilities, should be the basis for follow-up meetings led by security staff.

Documentation & Follow-up

Security's Role...

  1. Maintain audit trails for compliance and troubleshooting
  2. Lead follow-up meetings regarding security problems


Case Study

E-commerce Control
Retailer deploys change-control software for PCI compliance and security.

The Payment Card Industry Data Secu-rity Standard led Restoration Hardware to implement software to control changes on its e-commerce platform.

PCI requirements 10.5.5 and 11.5 call for deployment of file integrity monitoring software. In looking for ways to comply, Restoration Hardware--an upscale home furnishings retailer based in Corte Madera, Calif.--looked at several products but chose Solidcore S3 Control, which tracks changes in real time and enforces change policy.

"We needed to be compliant, so Solid-core allows us to satisfy the file integrity check portions of PCI," says Bobby Wen, manager of IT operations. "We got the security as a bonus."

The software prevents unregistered binaries from running, so it performs as an intrusion-detection and prevention system, he says. If an intruder manages to penetrate a system, he would not be able to upload a binary and hijack it. That security was what the retailer wanted, but the need to comply with PCI was ultimately what got the project funded, he says.

Solidcore easily integrated with the retailer's change management system, ControlTier, and ensures that change policy is followed, both from a timing and content perspective, Wen says.

Restoration Hardware deployed the software in its e-commerce platform, primarily in Web and application servers that handle credit card transactions. The software is comprised of agents installed on systems and a central console for administration.

Since Restoration Hardware was an early adopter of Solidcore, it had to do a little explaining of the new technology to auditors, who have since "blessed" the software as satisfying the PCI requirement, Wen says. With Solidcore's reporting functionality, the retailer can show auditors graphical views of system changes instead of simply showing them logs.

The company plans to expand use of the software beyond PCI compliance to Sarbanes-Oxley compliance.



For security teams
, there are a few points to keep in mind to avoid pitfalls with change management:

  • Understand the distinction between change management and configuration management. Change management is a process framework, aided by technology. Configuration management is more often equated with tools themselves, like a software agent on a server that monitors critical files for changes or a configuration database (see "Tools for Change," below). However, well-designed configuration management can greatly facilitate proper change management for security teams.
  • The biggest obstacles to achieving effective, risk-focused change management are people and processes, not technology. Focus on working with peers in different IT groups and business units to understand security's role in the change management workflow. "Security should become more of an 'active partner.' Tools should be less of a discussion than the policy and process creation," says Kris Brittain, a Gartner vice president.
  • When you evaluate configuration monitoring and implementation tools, look for those that can be integrated into organization-wide change management products. Don't create silos for security and security products.

Although configuration monitoring tools are beginning to integrate with change management products, convergence of the two is a ways off. For now, security professionals should focus on policies and processes, refine internal processes for risk analysis of proposed changes, and ensure that proper audit trails are kept of all steps in the change management workflow.


Tools for Change
Several products help streamline change and configuration management.

A variety of tools exist that perform centralized change and configuration management functions.

Some combine traditional patch management with configuration management options via agents or appliances that can inventory systems and applications, apply and monitor patch levels and control configuration details. These include BigFix Enterprise Suite, Configuresoft Enterprise Configuration Manager (ECM), Kace's KBOX Systems Management Appliance, BladeLogic Operations Manager and Lumension's PatchLink.

Other tools are more concerned with policy management and configuration monitoring, and perform file integrity monitoring and policy enforcement based on an organization's specific settings. Organizations define baseline configurations that map to policy, and once applied, the tools monitor for changes. These tools can either alert or automate configuration and remediation based on policy. These include Tripwire Enterprise, nCircle Configuration Compliance Manager and Solidcore S3 Control.

A third type includes traditional change management products that may integrate help-desk ticketing, change request and approval workflow tools, and built-in auditing functions. These include BMC Remedy, HP Change and Configuration Center, and IBM Tivoli Change and Configura-tion Management.


This was first published in May 2008

Dig deeper on Configuration Management Planning



Enjoy the benefits of Pro+ membership, learn more and join.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: