Unlock new pathways to network security architecture

Cover story: Want to shed appliances? Consolidation and new platforms hold promise for security teams.

In an interesting paradox, enterprise networks have experienced unprecedented sprawl and significant consolidation over the past 10 years. With new technology and application use at an all-time high, security teams require different ways to isolate, monitor and control traffic within their data centers and extended networks.

What network isolation and segmentation techniques are many companies now considering? How can consolidation and collapse of feature sets into unified platforms, and more condensed network security architecture at the perimeter secure sensitive data and corporate assets?

While security isn’t the primary driver of major network architecture overhauls, new threats are leading more organizations to re-architect portions of their networks. For some large organizations, the continued rise of devastating distributed denial-of-service (DDoS) attacks, embedded HTTPS control channels, and sophisticated malware may necessitate a redesign focused on network security architecture.

Business growth or operational changes can also increase the need to refresh network security architecture. These design changes are often coupled with equipment upgrades and replacement scenarios.
For many enterprises, compliance is the major driver for changes in both security and general IT operations. Any technology or internal design change that can limit or reduce the scope of the environment for compliance can save money and time, in years to come. Isolation of systems, applications, and network segments that handle payment card data, for example, can go a long way to limiting the scope of PCI DSS audits.

Isolation and segmentation techniques

Regardless of motivation, new considerations are driving the way networks are designed. In the past, many organizations used a traditional single or dual-firewall architecture that divided networks into segments at Layers 3 and 4, limiting IP address ranges and TCP/UDP ports that could traverse one segment or another. While this network security architecture is still the most common, more organizations are starting to control traffic at different layers and use emerging technologies that facilitate traffic capture, analysis and control.

Software-defined networking for monitoring and isolation: SDN is an emerging technology that implements network control through software and scripting in switches and centralized controllers. It’s heavily touted as a way to help security professionals implement access controls and traffic filtering, packet capture and monitoring, and isolation of traffic at Layers 2 and above. In March, Microsoft Principal Network Architect Rich Groves gave a talk describing the company’s use of the OpenFlow specification and commodity switch hardware to send large quantities of packet data to network monitoring devices (Figure 1). This same technique can easily be used to quarantine and isolate packets with specific attributes, potentially helping defeat DDoS and other attacks.

Openflow diagram
Figure 1. Software-defined networking based on the OpenFlow protocol.

Layer 2 isolation: While the use of virtual LANs (VLANs) to segment broadcast domains in a network is not new, more organizations are strategically using VLANs and private VLANs as a segmentation strategy for sensitive domains. Many newer switches, including Cisco Systems’ Nexus series and Juniper Networks’ EX devices, can also accommodate VLAN access control lists that allow for filtering based on MAC addresses and forwarding and capture of packets.

Isolation at virtual network layers: The use of virtual firewall appliances and newer virtual switches such as the Cisco Nexus 1000v, Juniper vGW line, and Open vSwitch is starting to emerge within converged infrastructure clusters as a sound isolation and segmentation practice. While most organizations aren’t replacing existing hardware-based security platforms with virtual systems, the use of virtual traffic control and monitoring systems is growing as a new layer of defense. Some of these systems offer capabilities that their hardware-based counterparts cannot (see tip on virtual networking).

Use of load balancers and content switches to isolate traffic: A majority of the traffic in enterprises today is HTTP, HTTPS or other application traffic. Load balancers and content switches are often used to provide availability and control for application traffic, but security teams can benefit from these technologies as well. While many leading manufacturers have offered security options in these products for some time (including port mirroring, scripting capabilities and DDoS defenses), security teams are starting to take advantage of these features as application traffic grows. Using application-layer packet attributes to direct and control traffic can help organizations isolate more sensitive or critical traffic, and identify malware command control channels using HTTP/HTTPS.

Internal VPNs and private cloud gateways: Several organizations have employed internal virtual private network (VPN) platforms to segment their networks. SSL VPNs can be easily set up and configured to act as a gateway to one or more segments of the environment, providing more robust authentication requirements, endpoint inspection capabilities, and integration with virtual desktop technologies. For organizations with private cloud deployments, new cloud “edge” gateways such as VMware’s vShield Edge or Juniper’s vGW can be installed to provide controlled access. Technologies such as VMware’s VXLAN allow migration and control of Layer 2 traffic across Layer 3 data center and cluster boundaries, which affords more flexibility to distributed virtual and cloud environments.

Unified platforms and condensed architecture

In addition to new isolation techniques and controls, organizations today are generally looking to collapse their infrastructure a bit more. The security community is actively using converged security appliances (often called universal threat management, or UTM systems) that offer a combination of services like antimalware defense, antispam and mail protection, content filtering, traditional Layer 3 and 4 firewall rules and even VPN and proxy capabilities, in some cases.

While most organizations aren’t replacing existing hardware-based security platforms with virtual systems, the use of virtual traffic control and monitoring systems is growing as a new layer of defense.

While these systems have steadily become prevalent and more mature, the technology is more viable for small to mid-sized businesses. Many enterprises are not sold on the technology, because it represents a single point of failure. It doesn’t support the scalability or performance required in large, fast (10 Gbps+) network environments. While this still holds true, many companies are looking to reduce the number of security layers within their networks and add enhanced functionality that may prove more effective at combating modern threats.

Over the last 10-15 years, many organizations followed popular trends in network security architecture, starting with the adoption of multiple layers of security traffic control points, such as firewalls. Some enterprises have even used technology from different vendors at each layer to prevent a single point of failure. This strategy may offer a multi-layered approach to network security, but it results in much higher implementation and operations costs, as well as overhead to manage these platforms.

Many enterprises use dedicated intrusion detection and prevention systems (IDS/IPS) to secure heavily used network segments and those that house sensitive data and applications. These segments often include the primary ingress points from the Internet, segments where a VPN connection terminates, and any exposed DMZ subnets, along with internal zones that need protection.

So what’s changing? Some Fortune 100 companies are replacing firewalls with next-generation firewall (NGFW) platforms. These systems offer more application and traffic behavior inspection along with new capabilities, such as user tracking from internal directory services and more robust protocol inspection. This strategy starts to approach the UTM concept, but with more capable and high-performing platforms.

Another major shift is the gradual consolidation of IDS/IPS platforms with next-generation devices and technologies. While a good number of organizations are still proponents of separate IDS/IPS, some companies are seeing benefits in using the NGFW platforms to handle both firewall and IPS functionality. As long as the performance of the network is not impacted with a single device handling so many security functions, this approach may make sense for some companies.

Planned upgrades and smaller zones

How should security and network teams proceed? First, align any network security architecture and monitoring changes with planned upgrades or changes whenever possible. If new or updated technology is already slated for purchase and implementation, investigate the access control, filtering and monitoring features built into these systems, regardless of vendor. If vendor selection and design phases have not been completed, suggest looking at technologies and designs that allow for the following:

  • Access controls and monitoring at Layers 2 and above: Instead of a consolidated firewall design, switches and other network devices may play more important roles in controlling and monitoring traffic, especially in widely distributed networks.
  • Integration with SDN protocols such as OpenFlow and sFlow: While many organizations may not be ready to make the switch to SDN just yet, preparing for it by purchasing equipment that allows for programmable functions and traffic control to be implemented is a sound idea.
  • Integration with virtualization and private cloud technologies from VMware, Microsoft, Citrix and others: Virtual appliance models with security technology are becoming available from numerous vendors. These systems can complement existing capabilities and network designs, especially in environments with virtual systems or a private cloud.
  • Application and protocol inspection: New types of NGFWs can either augment or potentially replace existing firewalls and IPS platforms.

Sizing up unified security platforms

Small and medium-sized businesses have adopted universal threat management devices more than enterprises. Trends that “stuck” for defense in depth are prevalent in many large organizations’ networks:

  • Multiple tiers of security access control/filtering devices
  • Different vendors (in some cases)
  • Separation of functionality

Another focal area for network and security managers is built on the concept of “compartmentalization” of network segments. With any redesign efforts, security teams should attempt to segment sensitive data, traffic and systems into more carefully controlled areas. While the concept of DMZs and network segmentation is not new, building more, smaller zones may make sense with a combination of VLANs, Layer 3 access controls and even application-level traffic monitoring and control. With advanced firewalls and new virtual platforms, this network security architecture is much easier to accomplish. NGFW systems and virtual appliances can help network and security teams lower costs, if they are replacing multiple platform types.

With new network technology and the availability of advanced security platforms, the design and architecture of many networks is likely to continue to change rapidly, in some cases, collapsing infrastructure with virtualization and cloud deployments.

Today, organizations are looking to collapse functionality into bigger, more capable platforms. Next-generation firewalls are starting to replace traditional Layer 3/4 firewalls and IDS/IPS at some Fortune 100 companies.

Dave Shackleford is owner and principal consultant at Voodoo Security, senior vice president of research and CTO at IANS, and a SANS analyst, instructor and course author. He is a VMware vExpert and has extensive experience designing and configuring secure virtualized infrastructures. He co-authored the first published course on virtualization security for the SANS Institute, serves on the board of directors at the SANS Technology Institute and helps lead the Atlanta chapter of the Cloud Security Alliance. 

Send comments on this feature article to [email protected].

This was last published in August 2013

Dig Deeper on Network device security: Appliances, firewalls and switches