Manage Learn to apply best practices and optimize your operations.

What ports should be opened and closed when IPsec filters are used?

In this Q&A, application security expert Michael Cobb explains how to set up separate branch IPsec filters that connect with a head office.

We are using a leased line to connect with branch offices. The branches' applications and services are terminal ones, Microsoft Outlook and print services. We want to set up IPsec filters for the branches so that they connect with the head office. Note that the setup is centralized, so there are no servers at the branches. How should we determine which ports should be opened or remain closed? What are some common mistakes that can be made during this type of evaluation?
Internet Protocol Security (IPsec) is a great protocol because it provides packet-level integrity, authentication and encryption. When correctly implemented, it can be a powerful and versatile part of a network's defenses.

IPsec also makes it possible to define who can use the services running on a server. So, for example, IPsec can be set to encrypt all Terminal Services traffic and check that each packet comes from authorized client computers and has not been modified in transit. To set up IPsec filters between your head office and branch network, you ideally need to test and validate them on a non-production server and workstation. Doing so ensures that the correct users can access the right services and that you haven't accidentally denied a service to an authorized user.

Although your application documentation should provide details of the ports and protocols they each use, it will still require careful testing to ensure all services can function correctly. For example, if a firewall separates your server and workstations, the firewall must have TCP ports 50 and 51 and UDP port 500 open to allow various IPsec and IKE (Internet Key Exchange) traffic through.

Another advantage of IPsec is that it enables a rule to be modified to only allow access from specific IP addresses. After all the desired protocols and ports used by your server's listening services have been selected and filtered, define the network's hosts or subnets that you want to allow them to connect to. Finally, create one last filter to deny all traffic that is not specifically allowed.

To test the filters, simply try to access your head office services from both the allowed subnets and the places where connections should be denied. To check that connections between branch office workstations and your main office servers are actually being encrypted, use the Windows IPsec monitoring tool, Ipsecmon.exe.

Start a connection from a client machine to the server and check that Ipsecmon.exe shows a connection in its monitoring window. The status indicator should show that IP security is enabled on the computer. If it does not, make sure the policy is assigned in the Group Policy Editor.

One definite improvement in Vista is the integration of firewall-filtering functions and IPsec protection settings. The capabilities make it far less likely that you will set up firewall filters that conflict with your IPsec policies. It's now possible to confirm, add, modify and delete firewall rules using a single snap-in called Windows Firewall with Advanced Security.

Finally, implement a defense-in-depth security model to protect the network. IP security filters should be just one aspect of an overarching defense. After all, they are only packet filters; they cannot prevent denial-of-service attacks or protect against an application exploit to a service that a filter allows.

More information:

  • Learn how to use the ipseccmd.exe monitoring tool in Windows Vista.
  • Mike Chapple explores the security risks of IPsec tunnels.
  • This was last published in March 2008

    Dig Deeper on Productivity apps and messaging security