Most network administrators are well-versed in preparing for software application or operating system upgrades. But what about hardware upgrades?
The spread of 64-bit technologies and the eventual shift to IPv6 will result in a major network infrastructure upgrade for most enterprises sometime in the not-too-distant future. Separately, there's always the danger of an unplanned or forced migration when a vendor goes bust or is taken over by a rival -- take the recent takeover of Nortel Networks Corp. by Avaya Inc., for example.
Once the choice to begin a network hardware upgrade has been made, the key to success will be open and honest communication and cooperation. The networking, security, financial and management teams, as well as the network's users, must all be involved and kept in the loop. Issues such as required changes to procedures and processes, test dates and any expected downtime need to be flagged as soon as possible to all affected.
Vendors and specialists in both internal and external applications should take part in the process, too. Differences over whether the right decision has been taken need to be put to one side to ensure the approved plan is implemented well. To help this process, the security team should highlight any concerns early on in the planning stage so that mitigating solutions can be agreed on. I would also recommend that a consistent vocabulary and terminology is established to avoid any confusion when discussing new technologies and equipment.
The challenge for the IT department will be to provide 24/7 availability while merging and integrating new equipment. Determining the complexity of a migration requires identifying and understanding the network's key functions and what devices and applications support the current environment, inventorying all hardware and software assets. This is an ideal time to audit current firewall and perimeter defense rule bases and highlight any risks that need to be mitigated. There are various rule-base analysis packages such as SecureTrack, Skybox and FireMon, which let you clean and apply your rule base to new devices. An open source option is FWDoc, which can produce an HTML summary of network and service objects, users, firewall rules and settings.
Once a new device is hardened and ready, it can be brought online as a secondary device to ensure it's functioning correctly. To ensure application components can still communicate with each other, connectivity and operability testing should focus on components, such as firewalls, servers and monitoring devices like IDSs, which could potentially be affected by the change. These tests are best performed using a script run before and after the upgrade to provide consistency. Gear like the ActiveSocket Network Communication Toolkit from Activexperts Software B.V.
is ideal for this task.
Logging should be set to "Log All" to help identify whether rules are still in use and to which applications they are relevant. Stakeholders of applications that may be affected should ensure support staff is available in the event of any problems and should provide feedback during tests. Testing should cover at least a full month so that all processes, such as month-end payroll, are tested prior to the change over.
It will be absolutely essential to test any encryption used within the system. Network protocols generally follow the big-endian format, but processor and hardware architectures use both so you may have to re-encrypt your data. If problems arise as a result of the upgrade that cannot be quickly resolved within the scheduled cutover time, you can activate your back out strategy by taking the new device off line for further investigation.
On the successful conclusion of the test period, the new device can be switched to be the active primary device, followed by post go-live testing and monitoring. You will have to ensure data isn't exposed during this process and that any controls mandated by regulation or accreditation, such as ISO 27001, are still relevant and effective. Document all of your security configuration settings so that you can provide assurance to your auditors that the appropriate controls are still in place.
A migration plan must build in overhead so that milestones are realistic and testing time doesn't have to be compromised when problems arise, which they undoubtedly will. Testing is a critical aspect of any upgrade, and unfortunately there aren't any shortcuts. Don't assume anything. You cannot batch-test several changes at once; if testing produces an unsatisfactory result, the root cause of the problem must be identified before going any further. This is much harder if you have installed several new pieces of hardware at once. Certainly, prior to any migration, it is necessary to verify that adequate backups exist and that backup and restore functions are working properly. These considerations and how to respond to them should be part of your contingency plan covering a worst-case scenario, such as a mission-critical application not working correctly after the new equipment is up and running.
To that end, when installing new equipment, there are plenty of integration questions. A useful source of help and guidance can be the relevant Internet discussion groups, where you'll find out others' experiences with a particular product. If you use open source software, this may be your only source of support if you encounter a major problem getting the software to function on the new network.
That's why, where possible, the initial rollout should begin with less critical systems and if they perform as expected, you can continue with the rollout until all systems are updated. This approach adds a further safeguard to the whole process. In the U.K., for example, the switch to digital TV is being done over a period of three years with little unexpected disruption. This is in contrast to the U.S. experience when the entire system was changed almost overnight.
Work together with your vendor
Most vendors offer tools, resources and planning materials to help with a migration or network hardware upgrade, often free of charge. You should certainly take advantage of any proof-of-concept offers whereby you can set up a proposed new system alongside the existing environment. Post-rollout vendor support is also an important factor; the need for go-live support should never be underestimated. If you're moving from a mainframe to a server farm, for example, do your administrators have the skills to manage and maintain the new equipment? One of the main advantages of mainframes is their reliability; will your server farm require more hands-on, 24x7 support?
If one of your key vendors is taken over, it is essential to quickly establish how existing contracts will be managed and how products and services are to be integrated. If product lines overlap, offerings will be rationalized and poor end-of-life support may mean you end up with no choice but to upgrade. But bigger doesn't necessarily mean better, and any newly merged company will have its own integration issues. Secure development lifecycle practices can get compromised. There's one large vendor I can think of whose security definitely dropped off after a series of purchases. So don't just mindlessly transition to new products; reevaluate which vendor has the best product for your environment.
Most vendors have a detailed road map that outlines planned upgrades, changes and product discontinuation dates. It makes sense then to be fully aware of the intended fate for your platform of choice. Knowing in advance that support for an older technology that's critical to one of your existing business applications is being phased out means you're not going to get caught by surprise. Downtime and breaches in data security are just a poorly planned migration away.
About the author:
Michael Cobb, CISSP-ISSAP is the founder and managing director of Cobweb Applications Ltd., a consultancy that offers IT training and support in data security and analysis. He co-authored the book IIS Security and has written numerous technical articles for leading IT publications. Mike is the guest instructor for several SearchSecurity.com Security Schools and, as a SearchSecurity.com site expert, answers user questions on application security and platform security.
This was first published in November 2009