IPsec also makes it possible to define who can use the services running on a server. So, for example, IPsec can be set to encrypt all Terminal Services traffic and check that each packet comes from authorized client computers and has not been modified in transit. To set up IPsec filters between your head office and branch network, you ideally need to test and validate them on a non-production server and workstation. Doing so ensures that the correct users can access the right services and that you haven't accidentally denied a service to an authorized user.
Although your application documentation should provide details of the ports and protocols they each use, it will still require careful testing to ensure all services can function correctly. For example, if a firewall separates your server and workstations, the firewall must have TCP ports 50 and 51 and UDP port 500 open to allow various IPsec and IKE (Internet Key Exchange) traffic through.
Another advantage of IPsec is that it enables a rule to be modified to only allow access from specific IP addresses. After all the desired protocols and ports used by your server's listening services have been selected and filtered, define the network's hosts or subnets that you want to allow them to connect to. Finally, create one last filter to deny all traffic that is not specifically allowed.
To test the filters, simply try to access your head office services from both the allowed subnets and the places where connections should be denied. To check that connections between branch office workstations and your main office servers are actually being encrypted, use the Windows IPsec monitoring tool, Ipsecmon.exe.
Start a connection from a client machine to the server and check that Ipsecmon.exe shows a connection in its monitoring window. The status indicator should show that IP security is enabled on the computer. If it does not, make sure the policy is assigned in the Group Policy Editor.
One definite improvement in Vista is the integration of firewall-filtering functions and IPsec protection settings. The capabilities make it far less likely that you will set up firewall filters that conflict with your IPsec policies. It's now possible to confirm, add, modify and delete firewall rules using a single snap-in called Windows Firewall with Advanced Security.
Finally, implement a defense-in-depth security model to protect the network. IP security filters should be just one aspect of an overarching defense. After all, they are only packet filters; they cannot prevent denial-of-service attacks or protect against an application exploit to a service that a filter allows.
This was first published in March 2008