Problem solve Get help with specific problems with your technologies, process and projects.

Troubleshooting proxy firewall connections

Investigating the TCP 'handshake' between clients and servers has always been a useful way to diagnose Web server and application problems. Firewalls, however, can interfere with the normal transmission control protocol process. In this tip, network security expert Mike Chapple explains how to diagnose client-server connectivity and security issues when a proxy firewall gets in the way.

With more proxy firewalls being placed between clients and servers, troubleshooting connectivity issues has become...

more complicated than in the past. Having a little knowledge of the Transmission Control Protocol (TCP) "handshake" process, however, can help diagnose connectivity problems involving proxy firewalls.

First, for those of us who've grown a little gray hair since our last introductory networking class, let's review how the TCP handshake works. TCP is a connection-oriented protocol; before any communication takes place between a pair of systems, a two-way tunnel must be opened up between them. This is accomplished through the use of special packets that contain flags. One such flag, the SYN flag, is used to denote a connection request, while the ACK flag acknowledges that a connection is open.

The complete three-way handshake is shown below:

First, the client sends a SYN packet to the server. Upon receiving it, the server composes a response packet. The server turns on the ACK flag to acknowledge the client's request to communicate. It then turns on the SYN flag to request a communication channel in the opposite direction: from the server to the client. When the client receives this SYN/ACK packet, it must acknowledge the server-client connection request by sending an ACK packet in response. At this point, the client and server may communicate in both directions.

However, proxy firewalls interfere with the normal TCP handshake by injecting themselves into the middle of the procedure, per the image below.

Clients connect to the proxy firewall, thinking they have established a connection with the server. The proxy firewall then opens a separate connection with the server, on behalf of the client. This unique approach provides an added layer of isolation between the client and server and allows the firewall to moderate the conversation, perhaps examining traffic at the application layer for potentially malicious content.

The proxy firewall's presence makes troubleshooting client/server connectivity issues a little bit trickier. In order to diagnose a problem, traffic needs to be monitored on both sides of the firewall. We need an external monitor looking at traffic on the outside of the firewall and an internal monitor connected to each internal interface of the firewall. Let's look at several common connectivity problems and how they'd appear in tcpdump output. We'll use the connection scenario explained above for all of our examples. Note that the Web server has two IP addresses: is the server's private address on the firewalled LAN, and is the public address exposed to the world.

Case 1: Server issue

Servers may experience connectivity problems due to denial of service attacks, misconfigurations, successful hacking attempts or other reasons. In these cases, because the firewall is functioning properly, there will be a completed three-way handshake on the outside of the firewall. The output will look similar to the one below:

19:24:36.959777 IP > S 735906036:735906036(0) win 16384
19:24:36.987938 IP > S 3607622985:3607622985(0) ack 735906037 win 8190
19:24:36.987961 IP > . ack 1 win 17640 

Notice that the SYN and ACK flags are bolded for your convenience. This is a proper three-way handshake. The problem will appear between the firewall and the Web server. If the server is down or otherwise not responding, the telltale sign will be a series of SYN packets from the firewall to the server, as shown below:

19:22:40.214672 IP > S 4182454:4182454(0) win 16384
19:22:43.154580 IP > S 4182454:4182454(0) win 16384
19:22:43.154580 IP > S 4182454:4182454(0) win 16384
19:22:43.154580 IP > S 4182454:4182454(0) win 16384
19:22:43.154580 IP > S 4182454:4182454(0) win 16384

Case 2: Firewall issue

Firewall problems may manifest themselves in different ways, depending upon the specific issue. If the firewall is completely down, there will be a series of unanswered SYN packets from the firewall to the server's public IP address. These will appear on the external monitor, and no related traffic will show on the internal one.

For more information

Can applications really establish a TCP connection without an open port? Mike Chapple explains what to believe.
Expert contributor Ed Skoudis reviews the tell-tale signs of a SYN flood.
Learn how to manage TCP ports.

On the other hand, if a firewall rule is configured incorrectly, external monitors may show a proper three-way handshake, and internal monitors may show no traffic at all.

Case 3: Application issue

If proper three-way handshakes are shown on both the internal and external monitors, firewall issues can generally be ruled out. In this case, the most likely culprit is a problem with the application itself. For example, an attacker may have compromised the application and changed its behavior, the service may be misconfigured, or the client may be making an invalid request.

These cases aren't all-inclusive; it's simply not possible to cover every possible troubleshooting scenario. That said, these examples provide an excellent starting point for diagnosing connection problems with a proxy firewall.

About the author:
Mike Chapple, CISA, CISSP, is an IT security professional with the University of Notre Dame. He previously served as an information security researcher with the National Security Agency and the U.S. Air Force. Mike is a frequent contributor to SearchSecurity, a technical editor for Information Security magazine and the author of several information security titles, including the CISSP Prep Guide and Information Security Illuminated.

This was last published in July 2007

Dig Deeper on IPv6 security and network protocols security