DISPATCHES Cutting costs was the only way to keep United Airlines flying high. Rich Perez's answer was to rebuild
|United Security Taking Off|
United Airlines wasn't necessarily about to be grounded by an inefficient IT network. A stagnant economy, rising fuel and labor costs, competition from discount carriers and the 9/11 fallout were doing a good job of that on their own. Yet, security manager Rich Perez had his mandate: Rearchitect on a shoestring budget and make the network more secure.
The project, which Perez launched in early 2004 after a four-month planning session with the airline's business units, ran in concert with an overall evaluation of United's cost structure. Nothing was left to chance as the Chicago-based giant put every expense on the table--flight schedules and routes, food services, baggage handling, fleet maintenance, ticket sales, marketing and promotions. The overall conclusion: United was a victim of its own fragmented organization.
Redundancies had to be eliminated and resources reused wherever appropriate. United's corporate flow chart made this even more daunting. United wasn't one large company, but rather a conglomerate of many small companies that provide services to the airline. From an IT perspective, each division acted like an independent company, complete with its own management and operational infrastructure. United.com, for example, had a different IT network structure than MyPoints.com. The same went for amenities services, the Silver Wings product line, the Mileage Plus program, the Star Alliance management and all overseas operations.
"The airline industry is staggering right now. Everything is being scrutinized more than ever. Every dollar is examined and cross-examined," Perez says.
"If we present a case, it really has to make sense both financially and functionally. The goals and the intents have to be crystal clear."
Consolidating the small service companies into United's corporate structure would eliminate redundant layers of management and save millions of dollars in operating expenses. Maintaining that efficiency, Perez found, would require a continual examination and adjustment of the network architecture. Without secure IT, United would cease to fly.
Before taking off on the project, Perez cleaned out United's IT cabin, inventorying the infrastructure and assessing every security apparatus, application, service and policy.
As it turned out, most things weren't working well.
The most obvious weakness was the security provided by United's collocation service, which managed many of its e-commerce applications and Web-based services. "It provided security, but it was a generic model that responded more to its own requirements than the particular needs of United Airlines," Perez says.
Perez's team drilled down into every aspect of United's security. They applied broad assessment criteria, scrutinizing flexibility, functionality, scalability and--perhaps most important--the costs, both upfront and ongoing. If something was working, it would be kept and possibly deployed in other departments. If it wasn't working, it would be jettisoned.
United's precarious financial situation meant that there wasn't a lot of money for new purchases, big product deployments and staff education. Perez had to reuse as much of his existing infrastructure as possible.
"Reusage is one of the most critical factors in revising a network's architecture," he says. "This becomes especially true when you are in a financial crunch and you need to squeeze blood from the rock. If something only fits in one environment and cannot be leveraged anywhere else, it's not a solution--it's a problem."
Security experts agree, but they're quick to add a word of caution: Reusage is only a starting point. Threats evolve, solutions change and rearchitecting offers an ideal opportunity to augment existing solutions with new, innovative technology.
"It's a reasonable first pass to throw out everything that's stupid," says Eddie Schwartz, an independent security consultant. "But, having said that, it's becoming more complex to protect what becomes a more open perimeter.
So, you need more complex technology that you can embed within your existing technologies."
Only after Perez exhausted the useful pieces of his existing infrastructure did he fill the gaps with new investments.
"We realized that if we were going to consolidate and still match or exceed performance, this was the time to go in some new directions," Perez says. A transition plan for phasing in new technology was roughed out, based on feedback from United's business units and IT department. Representatives from United's pilot and flight attendant organizations also took part, as did several airport reps and the company's internal corporate audience. A plan was built around the business needs of each United appendage and the technology requirements put forth by IT.
The assessments showed that divisions using a Cisco Systems IDS were having the greatest success at detecting attacks and malicious activity. Perez decided to rip out the myriad other IDS solutions and standardize on Cisco-Works Security Information Management Solution v2.2, believing United could replicate the product's success across all divisions while driving down costs through decreased training and maintenance expenses.
Performance testing was conducted on baseline legacy data, and comparison analyses were done on throughput and security levels as new systems were implemented.
Similarly, Perez found that United was getting a lot out of its software security solutions; supporting the hardware that these solutions resided on was the problem. To simplify the network and reduce hardware costs, United turned to Crossbeam Systems' high-performance security appliances for hosting Trend Micro's AV and content filtering, as well as Secure Computing's SmartFilter URL filters. The cost savings were apparent from the start, and United's security improved.
Choices like these helped Perez make the financial case. "The hardware investment helped us consolidate and leverage our existing investment in software."
Perez believes United will further reduce security costs through less-expensive support and maintenance contracts. Less training and staff support will be required, since everyone will be using the same equipment.
Consolidation, however, wasn't always the answer. United was trying to cut as much cost as possible, and if that meant leaving multiple, cheap software programs running where one expensive one would do, Perez's hands were tied. For example, United ran three separate software packages to handle URL content filtering and antivirus.
In particular business units, however, United standardized on Aladdin's eSafe product suite.
As a major buyer, the airline was able to gain considerable negotiating leverage when it went shopping for a single-source solution to handle both functions. However, if the Trend Micro and Secure Computing combination worked and was cost effective for a particular unit, it stayed.
"We had to justify everything we did through cost-savings," Perez says. "If it cost more money than the present model, we weren't doing it."
Virtually First Class
Perez admits that there are trade-offs with any rearchitecting project. On one hand, a consolidated system gives the IT team centralized control over security. The flip side, of course, is that the single centralized system gives hackers and malware writers a bigger target. For all its faults, a fragmented, heterogeneous infrastructure at least limits the damage an intruder can do. "If you do compromise one thing, you don't get the whole store in a diverse infrastructure," says Gary Morse, president of security consultancy Razorpoint Security Technologies.
The key to securing a standardized network is in making a distinction between "fragmentation" and "segmentation." Fragmentation leaves pieces scattered about; segmentation brings everything under one roof, but builds partitions between different business units and information repositories to preserve integrity.
United decided that virtualization was the best way to segment its new network because it provided the same effect as physical separation, but without the expense of purchasing and deploying new hardware.
"As we consolidated, we took a lot of these infrastructures and kept them logically segmented through virtual technology, even if they were physically all one unit," Perez says. "With virtualization, we can build in a lot of diversity within an infrastructure while still creating a lot of resilience. This way, we avoided the 'all your eggs in one basket' scenario."
In such architectures, legacy divisions and interdepartmental barriers are preserved with virtual partitions. Behind the scenes, data is discretely divided. Yet, everything exists within a unified framework that's easier to manage and less costly to maintain.
"The fewer doors, the fewer that you have to lock," Perez says.
Moreover, the changes are essentially transparent to users. Avoiding huge cultural changes was every bit as important to Perez as any adjustments made in terms of hardware and software. He wanted a seamless transition, reducing complexity and increasing usability. The magic of security would remain behind the scenes.
After a year of work, Perez has achieved many of his goals: a consolidated security infrastructure, a hardened network that is easier to maintain and reduced enterprise-wide operating expenses.
Perez projects an initial savings of $2.5 million through the reduction of existing maintenance and support charges and resource distribution, among other expenses. By the time the project wraps in early 2006, United will have consolidated 22 security application servers into four in each of its 10 data centers nationwide, while supporting as many as 7,000 users.
According to Perez, the savings will be ongoing, but it's too early to project an ROI because the project's overall cost isn't clear.
"With more eyes, we have been able to see a lot of things that we never saw before," Perez says. "We really are able to provide better security now, with more people examining the logic of it."
Ironically, though, the restructuring hasn't reduced United's security head count or spending. In fact, Perez expects to add more people as United continues to address the need for compliance with regulations, such as Sarbanes-Oxley.
Perez believes that the rearchitecture project will actually keep head count far lower than if the airline had to support the decentralized infrastructure.
What he learned along the way, though, is that rearchitecting for security isn't a once-in-a-lifetime event or something forced through a crisis. Reassessing and adjusting network architecture is a continual process, and networks and security technologies are constantly changing. Only through periodic re-examination of the infrastructure and technologies will enterprises maintain efficiency and effectiveness.
"We shouldn't ever be resting on our laurels," Perez says. "We should constantly be re-evaluating and reassessing our environment. We should never get complacent."