Diana Kelley and Ed Moyle
Published: 05 Nov 2009
Networks and data centers are in constant flux. In large organizations, it's rare for a day to go by where we don't...
need to provision new servers or users, upgrade critical systems, patch databases, change firewall rules or update Web applications. But while change is a natural and necessary part of network growth, unmanaged change can quickly lead to chaos that exposes critical data and resources to attack. To prevent exposure, change management systems and procedures are implemented to help companies ensure that policies and approvals are met before a change to the system is implemented.
But simply putting a change management process in place is not sufficient. As Pete Lindstrom, research director of Spire Security explains, "If you are creating a change management process because you don't want anything to change, you are missing the point." Rigid, poorly implemented change management processes can be perceived as gating factors covered in red tape that slow growth and impede work. Done correctly, change management integrates seamlessly into the overarching risk management program and can decrease approval times for critical changes and keep the company moving forward. It can also reduce the response time when problems occur. As Jim Ferguson, an IT manager based at Northwest Hospital and Medical Center in Seattle and an employee of Siemens Healthcare, the managed services provider that manages the medical center's IT environment, observes, "When there's a problem, the first question is 'what changed?' Now we can go back and look at what changes were made."
CONSEQUENCES OF CIRCUMVENTING CHANGE MANAGEMENT
So change management is good, right? Why doesn't everyone do it? Because sometimes following the rules seems like a real waste of time. Ever missed a connecting flight and been told to wait in a long line so you can re-book with the airline's customer service representative? As the line of angry flyers serpentines slowly forward, it's not uncommon to hear groans of frustration as travelers peruse their smartphone Web browsers, seeing perfectly good flights available but that can't be booked except by the service rep at the counter. As experienced travelers we know it would be faster for us to manage the re-booking ourselves, but the airline's change management policy forbids this.
Trying to get things done at work can feel equally frustrating. Take for example a team of developers that has just written a much better looking and easier to use version of the organization's website. The only problem is that it can't be tested by the remote QA team because the firewall is blocking access. Waiting for change management approval could take weeks, so as the firewall admin, it may be very tempting to want to help out the development team by temporarily opening a port on the firewall. Sadly, we've all seen some variation of how that story ends: a worm or Trojan is introduced onto the internal network, a sniffer is planted on a server and credentials are stolen, or a previously protected database is exposed to attackers.
Many of the exposures associated with lack of change management are more complex and subtle than in the example. This is due to the complex nature of today's network environments. Networks are complicated ecosystems and dependencies are not always clear, especially to someone who only sees part of the whole system at a time. A database administrator changing an IP address could lead to a critical service outage. A router administrator that configures a new static route may inadvertently redirect or block traffic from hundreds of remote offices.
The purpose of change management is to prevent unintended consequences, such as the ones described, and ensure that changes or alterations to systems are implemented according to an approved framework or model. That's not something many employees would argue with. The problem occurs when an employee, such as the firewall admin in our example above, thinks that circumventing the system will allow things to work more efficiently--or feels that following the processes somehow detracts from getting "real work" done. So the challenge is not simply putting change management in place, but also gaining buy-in from all users of the system so that they're incented to follow the change management process rather than circumvent it.
LAYING THE GROUNDWORK FOR EFFECTIVE CHANGE MANAGEMENT
Whether your organization is just starting to build a change management program or is in the processing of improving and fine-tuning an existing one, it's important to focus the program around business improvement and maximum awareness. In the strictest IT sense of the word "change" refers to actual changes to a system or application such as a new sub-domain, firewall rule, or addition to a CMDB (configuration management database). As an integrative component in risk management, change extends to include all ramifications resulting from a particular change. Lindstrom explains, "The important point here is to forego the 'ready, aim, fire' mentality without implementing a monolithic process that is unmanageable and easily circumvented."
For example, how much downtime will be incurred? What is the cost associated with the change and any resulting downtime? Will the change impact other services? What is the schedule for the change and will it conflict with any other changes? The ability to answer these questions depends on a number of factors including the assessors' knowledge of the systems and operations in place as well as the tools and processes the organization has put in place to help manage change. For example, while a ticketing system such as Remedy can help provide information about what changes are in the queue, a shared calendar helps prevent scheduling conflicts.
The team responsible for the Unified Compliance Framework and "The Change Management Toolkit" sums this concept up nicely: "the objective of change management is to enable beneficial changes to be made with minimum disruption to services. . . Therefore, what you want to ensure through change management is that all changes are known." Ferguson recommends reading, "The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps," for specifics on building an effective change management program. Says Ferguson, "Of all the things you want to do with ITIL, the first thing to really look at is your change management," he says. "Organizations that invest in change management see a lot of benefit in reduced downtime and fewer outages."
Users report that as their change management programs mature, the scope of the program often expands. But this doesn't always translate to having complete control over every piece of equipment in an environment. Rather than trying to put all devices under the auspices of the IT department's change management program, try to understand what benefits can be offered to the device or service owner to encourage more active participation. For devices that can't be included in the change management program, maintain protection from other parts of the network with approved security controls such as zoning, intrusion protection and antimalware.
As repeatable standards and measures are put into place, change and risk related questions can be answered in a more efficient and normalized manner. At Northwest Hospital there is a workflow process known as pre-approved changes; routine changes that happen over and over. "We perform change management on these routine changes, but we simplify the approval process each time," Ferguson says. Some changes may, eventually, be considered routine and well known enough that they can be accepted outside of the formal approval process.
David Sherry, CISO for Brown University in Rhode Island notes that centralizing servers and services in a managed data center increases the number of known changes that can be performed without formal approval, which, in turn, serves as a motivation for departments and users to opt-in to the centrally managed data center offerings. "Moving to the data center has so many benefits that most departments want to do it voluntarily," says Sherry.
CHANGE MANAGEMENT BUY-IN AND ENFORCEMENT
Getting buy-in for the change management process is a much more successful strategy than using an audit or compliance "stick" to force participation. Not every vertical has a big compliance stick to yield, but even change management gurus in the heavily regulated financial services industry report that "carrots" are most effective. And the tastiest carrot is wrapped in a tangible business gain. As Ferguson explains, "Sometimes marketing change management to staff is difficult. It has to be marketed as beneficial."
As most security professionals know, selling "better security" isn't always the most motivating benefit for many users. Sherry cites a few positive ways his team has promoted buy-in.
"Change management can be a great communications vehicle, encouraging groups from different departments to work together," he says. Open communication can lead to smoother functionality too--he recalls one meeting where a team had seen a new firewall installation scheduled on the shared calendar and they came to the next change management review meeting with a comprehensive list of tests that would need to be performed "to ensure everything was running smoothly, and there was no loss of business continuity." Echoing the business continuity benefit, Ferguson points out similar benefits, "Without knowing what's causing a suspicious event, we have to treat them all as critical."
Another way to encourage participation is to make it easy for people to engage with the process. Make sure that the procedures are documented clearly and easy to find on an accessible website. If your organization is large and distributed, create a central landing page for the baseline change management procedures and branch as needed to sub-procedures that apply to business units and subsidiaries, different geographic locations, or departments.
Regular change management meetings were recommended by all interviewees. Both Northwest Hospital and Brown University have change management review committees that meet every week to review requests. This allows employees to plan change requests and reduce "last minute/just this once" workarounds. Of course, there are always going to be special requests for expedited approval. Make sure your system is flexible enough to allow for these exceptions. At Brown, there is a special line of ticketing that routes the request to key approval personnel who can approve the change and process it quickly, says Sherry. When looking at ways to reduce the approval timeline, Lindstrom says, "Change is continuous in any organization and your goals should be to make changes as quickly as you can within the realm of practical controls."
A final recommendation--automate wherever possible. Allow users to enter in their own change requests at a self-service site that feeds directly into the main ticketing and change management system. Configure the site on a wizard-type model that makes it easy for users to enter the most common requests, like new user provisioning and Web application updates.
CONTINUOUSLY ASSESS AND IMPROVE CHANGE MANAGEMENT
Over time, organizations will find it helpful to fine-tune their change management programs. As the change management administrator, Ferguson is responsible "for looking for improvement opportunities." Sometimes these opportunities are identified investigating failure points and determining whether they were caused by a lack of change management.
Auditing or performing assessments on your change management program is a useful way to measure coverage and effectiveness. Review the approval process and determine helpful metrics such as: Number of exceptions over time, are they increasing or decreasing? Or number of failures, how many unapproved/unplanned changes resulted in downtime over a given timeframe? Approval time may be streamlined by tuning the accountability flow and ensuring key approvers have time built into their day to review requests and coordinate responses. Another area for improvement is increased automation via integration with systems such as security information and event managers for better visibility into event root analysis and reduction in false positives during scheduled downtimes.
For additional guidance on change management assessment, ISACA has a useful publication entitled "Beyond Checklists: A Socratic Approach to Building a Sustainable Change Auditing Practice" that lays out a methodology for testing change management effectiveness on the basis of the 3 C's (culture, controls, and credibility.) And the Unified Compliance Framework recommends auditing change management in these seven areas:
- Policies and Procedures
- Tools and Automation
- Skills and Expertise
- Responsibility and Accountability
As David Sherry says, change management is "absolutely necessary. It promotes standards, process improvement, reduces complexity and risk and provides sanity in complex environments." A small investment in change management can reap a great reward in disaster prevention. To benefit from the control and insight change management brings, organizations don't need to implement fully mature systems or spend a lot of money on expensive vendor solutions. But they do need to get a process in place and to make sure it's user-friendly enough that employees will opt-in rather than work-a-round.
The important thing is to, as Jim Ferguson advises, "Do something. Something is better than nothing. You don't need a huge, big investment. The real benefit is in defining a process and following that process."
Diana Kelley is founder of consultancy SecurityCurve and former vice president and service director for the security and risk management strategies service at Burton Group.
Ed Moyle is currently a manager with CTG's Information Security Solutions practice, providing strategy, consulting, and solutions to clients worldwide as well as a founding partner of SecurityCurve.
Send comments on this article to email@example.com.