Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Data Encryption and IDS / IPS: Getting a better view of network activity

Encryption can to help secure data and meet HIPPA requirements, but the technology blocks sight of network activity by blinding IDSes and IPSes. Learn how to have an effective encryption and IDS/IPSe solution simultaneously.

Encryption may be good for securing data, but it blinds network-based IDSes. While there aren't any surefire fixes, these techniques will steer you in the right direction.

Encryption used to be unequivocally good for security. After all, it kept the bad guys from getting at our private information, right? Who could argue with that? Many of us became crypto-evangelists, demanding encryption everywhere.

Then we realized that we were blocking our view with all this encryption.

Administrators need to be able to look at the entire network to properly defend it. IDSes, IPSes, sniffer tools and network analyzers provide a clear view of network activity. But, the more prevalent encryption becomes, the more we lose that view.

So is encryption bad for security? Are IDSes and IPSes dead? No, but imprudent use of encryption can send a well-meaning network security engineer into unsafe territory.

The pressure is on for enterprises to implement encryp- tion as a standard of due care--HIPAA mandates privacy for health care transactions, and California's SB 1386 requires the disclosure of security breaches of unencrypted personal information. In the private sector, the Payment Card Industry (PCI) security standard includes rules regarding data encryption for credit card transactions.

How can an enterprise protect itself from the bad guys, meet regulatory requirements and have an effective IDS/ IPS solution? While there are no clear-cut answers, we'll explore some of the ways you can keep a clear look out for hazards on the horizon.

No Sure Fix
Encryption may seem like the perfect security fix, but it doesn't protect against everything.

TRIVIAL PASSWORDS A password can be guessed over an encrypted channel just as easily as over a cleartext one.

APPLICATION-LEVEL INJECTION ATTACKS Once an encrypted channel is established, the application is no less vulnerable because of the encryption.

BUFFER OVERFLOW OR SYNTATIC EXPLOITS If they occur before or during the login process, your enterprise isn't covered.

CERTAIN DoS and DDoS ATTACKS Resource starvation attacks will still work, as will some attacks on the crypto protocol itself.

SOCIAL ENGINEERING ATTEMPTS Technology can't protect against attacks on the user.

This is not an exhaustive threat list. Also, your encryption software may have security problems of its own. Do a quick search on critical security problems found in OpenSSH or OpenSSL to check. Nothing is without risk, not even security software.

--Dorian Deane & Benny Jones

Fork in the Road Fundamentally, encryption blinds your IDSes and IPSes. Regardless of whether the IDS works through signature matching or anomaly detection, it needs to see the packet in cleartext to detect most attacks. But without encryption, confidentiality (one of the cornerstones of good information security) is reduced, if not completely lost.

While there are solutions that come close to solving parts of this problem, a balancing act is often required. Sometimes the only answer is to weigh the competing needs and pick encryption or IDS--not both.

When evaluating the options, you need to understand the importance of data privacy in your network environment. If the data is on a company's public Web site, the sensitivity is low and privacy may not be as important. However, if the data is sensitive financial information, privacy becomes paramount and encryption is likely to be one of the few practical risk mitigations.

Still, security decisions rarely hinge solely on encryption versus cleartext. If the data must be encrypted and throwing away your IDS is unacceptable, alternatives may emerge from a threat analysis. Do you need to encrypt all the data, or just certain fields? Is the encryption intended for privacy, to prevent injection attacks, or both? Will encryption on the wire make application layer attacks more difficult?

Different solutions will present themselves depending on the threats. For example, encryption is effective at reducing threats such as TCP or UDP injection and spoofing attacks. Blindly spoofing one end of a TCP session is hard; the addition of well-managed encryption makes it nearly impossible. However, encryption is useless against other types of threats (see "No Sure Fix," at right).

Down Shifting
The ever-increasing number of threats to internal networks has caused a virtual stampede to encrypt corporate networks from end to end. This is laudable, but, again, we blindfold our IDS in doing so. One common compromise is to limit the use of encryption to only the less trusted portions of the path. A typical technique is to decrypt at trust boundaries (e.g., using load balancers, VPN servers, or some other front-end proxy) and resend the traffic in the clear over the more trusted network.

This option may be acceptable when the load balancers and the endpoint servers are close, such as on the same VLAN. But as enterprise networks become larger, these edge networks become less trusted by virtue of the size of the internal networks from which they are managed. Moreover, requiring that proxies or other endpoint devices be physically close can be a design constraint. Re-encrypting the traffic before putting it back on the wire can impose significant processing requirements, but is often found in reverse-proxy architectures.

Encryption of the data itself rather than the entire communication session is another alternative. Generally, this can be decided on a case-by-case basis. The simplest example is the use of anonymous FTP to send an already-encrypted file. While FTP offers no encryption, the file itself can be encrypted beforehand. Similarly, database transactions can be sufficiently secured by encrypting only sensitive fields.

Keep in mind that a threat analysis in the context of the existing architecture may reveal other considerations. In the case of FTP, a data channel is opened that may require firewall exposures in some environments--this would clearly be a factor against using FTP above and beyond the IDS problem.

SSL Keys If the primary threat is a non-specific inbound attack against a Web server, sharing the private SSL key of the Web server with the IDS could work; the IDS can watch the session key negotiations and decrypt the SSL streams. The benefit is clear: Attacks that may have been hidden within the SSL stream are visible to the IDS.

While this can be a workable solution, it puts the private key at somewhat more risk since it's now available on at least two hosts--the Web server and the IDS--each with its own set of vulnerabilities that must be mitigated. In some architectures, this approach can also overload an IDS, since it now carries the added burden of decrypting traffic as well as performing its normal signature checks.

While this is mostly a capacity-planning problem, it's an important one--if a key-negotiation packet is dropped during the SSL handshake, the session will be essentially invisible to the IDS. Another drawback is that this solution is not widely available for protocols other than SSL.

IPSec AH If the main threat is an injection or replay attack against the network connection itself, a particular form of IPSec may offer sufficient protection. Although IPSec is commonly thought to provide encryption, only the Encapsulating Secu-rity Payload (ESP) mode of IPSec actually encrypts (see below).

The other mode, IPSec Authentication Headers (AHs), doesn't encrypt the data. Instead, it uses cryptographic checksums to authenticate each packet, which prevents injection and replay attacks while leaving the data portion visible to the IDS. Although IPSec AH does nothing to protect confidentiality, since the payload remains in cleartext, it may provide an easy solution for handling less sensitive data.

IPSec ESP IPSec ESP can provide cover for an application that doesn't use encryption; that is, all communication between two IPSec endpoints is encrypted. In some cases, this is a great solution, but there can be drawbacks.

With IPSec ESP, the choice is binary--you either allow or disallow the IPSec traffic. IPSec, by its layer 3 nature, encrypts everything; no network IDS is able to peer through the crypto-armor to see what's going on inside. Thus, if your network firewall allows the IPSec traffic, it must also allow all possible TCP and UDP services between the IPSec endpoints.

Network firewalls simply cannot see inside to allow or deny specific services. And, since it is nearly impossible to configure a functional system without several listening services above and beyond the minimum you want to provide, these un-wanted service ports are exposed through the firewall.

The good news is that endpoint controls, such as host firewalls, can help re-duce those problems. Most host firewalls can apply their policies after decryption to filter the undesired ports; and, more to the point, host-based IDSes can see again.

End of the Road
While there are no monolithic solutions to the rather vast scope of this issue, endpoint security is likely to be at least part of everyone's solution.

The history of security has shown a trend of ever-shrinking perimeters, and the natural progression of the security perimeter is to move inside to the hosts themselves. This is particularly true of firewall technology: It was once sufficient to protect the perimeter of the enterprise; later it was sufficient to have zones of higher security inside the corporate perimeter; and now, firewalls are often installed directly on the hosts.

The same logic applies to IDS technology. End-to-end encryption is desirable, and, by applying IDS checks at the endpoint, it is less critical that a network IDS see the traffic in midstream.

An endpoint IDS solves several fundamental problems because it can look at the packets after decryption, thus largely avoiding the blind IDS problem. An oft-mentioned drawback is that if something does get past the IDS, it's already on the host--ideally, you'd like to get a warning while the intruder is still scratching at the front door.

This problem also plagues network IDSes because, in practice, there is so much low-level attack noise that it drowns out warnings of a more determined attack. Host IDSes suffer far less from background noise. Also, the detection of a scan behind the firewall, unless it comes from an authorized source, will draw an administrator's attention. At the network perimeter, the same scan would likely go unnoticed.

The biggest problem with moving security to the endpoints may be more social than technical. Few organizations are staffed or empowered to effect a lot of oversight at the level of the individual host; their domain consists of the wires and the packets that run over the wires.

One solution may be for IT workers who are closer to the application's requirements to take responsibility for deciding specific IDS rules for hosts while security departments mandate a baseline for host IDS signatures. The security department then takes on an oversight role for rules added locally by the application owners.

Road Work
With perimeters shrinking, applications fighting their way through firewalls and IDS efficacy being hampered by a pandemic of false positives, organizations are evolving to respond to these trends, but security remains a balancing act. What's important to remember is that encryption carries its own set of risks. Preservation of confidentiality is nearly always a desired goal, but it is not free. And, the cost isn't just a few extra CPU cycles to encrypt a packet--it can be as high as a crippled IDS and a well-hidden attack vector into the network.

We aren't just hiding network activity from the bad guys--we are hiding it from ourselves. While encryption remains one of the key pillars of information security, network and security engineers need to understand the consequences of its usage and find ways to ensure that it doesn't become a roadblock.


Dig Deeper on Email and Messaging Threats-Information Security Threats