Filling SMTP gaps -- The secrets to using e-mail standards

Learn how to use TLS and S/MIME to strengthen your e-mail security.

by Joel Snyder

SMTP-based e-mail has long been considered insecure, and for good reason. The basic protocol doesn't include encryption, authentication or data integrity services. However, there are standards-based ways to increase the security of SMTP-based e-mail. In this article, I'm going to discuss how to add encryption to the SMTP protocol, as well as how to add data integrity, sender authentication and encryption to e-mail messages -- all using standards that have been available since 1999!

Although SMTP itself is a fairly simple protocol, the idea of extending SMTP has been around for a long time. The mechanism to extend SMTP is a general one, and many different extensions have been defined to extend and stretch the capabilities of one of the Internet's oldest protocols.

Start with TLS

One of the extensions within SMTP is the addition of the TLS (Transport Layer Security) encryption and authentication protocol. When a client and server that support TLS talk to each other, they can encrypt the data channel and thus guard against eavesdroppers.

TLS is the IETF-standardized version of the SSLv3 protocol, which is widely used across the Internet. All popular Web browsers support SSL and TLS. SSL is the protocol that is negotiated when you use an "https:" URL. The differences between SSLv3 and TLS are not important, and most SMTP servers also support clients that want to talk SSLv3 instead of TLS. In this article, I'll only refer to TLS to simplify the discussion.

With TLS, the server side of the transaction sends down a digital certificate that is used to prove the server's identity. Optionally, the SMTP client can send up a digital certificate to prove the client's identity. Once the server's certificate is sent down, the client creates an encryption key, encrypts the key using the certificate of the server and ships the encrypted key up to the server. The client and server then start up an encrypted channel, and everything from that point forward is protected against eavesdroppers. This is how encryption works on the Internet with https: Web pages, and this is also the way it can work if you've enabled TLS support in your SMTP e-mail server. (I am glossing over a lot of the potential variations in TLS that are allowed, but the cases that I'm describing represent 99% of the TLS use on the Internet today.)

Using TLS with your SMTP mail server is generally a simple matter. All modern e-mail servers fully support TLS and have for many years. More importantly, TLS and SMTP are extremely interoperable. My company has been using TLS on our mail server since early 1999. Although we used to run into the occasional domain our mail server couldn't send out to, this hasn't happened in several years. In fact, almost all of the problems we encountered in TLS interoperability were caused by defective firewalls with poorly written SMTP protective code. Very old versions of WatchGuard's Firebox and Cisco's PIX had this problem, for example.

How to use TLS

To make use of TLS with your mail server, you've got to do two things: turn the feature on and get a digital certificate. How to enable TLS is beyond the scope of this article, but if you search for TLS or STARTTLS (the name of the SMTP service extension that enables TLS) in your mail server documentation, you'll probably find it very quickly. Getting a digital certificate is a little harder. In fact, this is probably why most people haven't turned on TLS -- they get to this step and freeze up. Fortunately, it's become a lot easier to get a digital certificate than it used to be. You can give Verisign a couple of hundred dollars and wait a week or so, but there are many options for getting an inexpensive digital certificate signed by a well-recognized certification authority, such as RegisterFly, within a few minutes.

In fact, you don't have to pay for a digital certificate to run TLS in your mail server. For testing purposes, and perhaps even for production, you can use a self-signed certificate. Your SMTP server might even have a built-in command to generate a private key and matching digital certificate. CAcert will also generate signed certificates for you for free.

With TLS enabled, all of your e-mail traffic from your company's SMTP mail server to external TLS-enabled SMTP mail servers will be encrypted, from start to finish. If you are using a POP or IMAP e-mail client that supports TLS -- and all modern ones do -- when you send mail from your client to the mail server, it'll be encrypted.

Although TLS includes server and client identity, these are fairly vague concepts when it comes to SMTP-over-TLS. The binding between a server's identity and the identity in the digital certificate is a loose one. There is no clear and obvious standard for mapping the identity I want to talk to with the identity in the certificate. In addition, it's not clear to the SMTP client or server what action to take if the digital certificate doesn't seem to match. On the Web, a dialog box pops up and the user makes the decision whether or not to accept the certificate. A client and server have no one to ask when things don't look right.

The result of the TLS transaction is perfectly fine encryption, but rather weak authentication. If you don't match the server certificate to the identity you think you're sending to, you're susceptible to a man-in-the-middle attack. You could be sending encrypted traffic, but not to your intended recipient. Since we're accustomed to SMTP mail being forwarded in an uncontrolled manner through a number of different relays, all we're really doing is adding encryption against casual eavesdroppers. Not a huge benefit, but at virtually no cost.

All of this comes down to TLS not being the ultimate answer for e-mail encryption. But like any good security system, it's a question of layers. Throw TLS on, and you're no worse off then you were before; in fact, you'll discover that about 10% of your e-mail is encrypted. For an investment of a few hours of time, you're adding another layer of security.

There are some cases where TLS is critical. If you are supporting any sort of Internet, SMTP, POP and/or IMAP service, then TLS encryption on these services is a security requirement. Otherwise, you're sending passwords out in the clear when you authenticate people coming in to your mail service. Many companies are turning to SSL VPN appliances from companies such as Juniper and Nokia to front-end their SMTP, POP and IMAP services with TLS encryption.

Sophisticated mail systems often can be configured to be "pickier" about TLS services when talking to trusted partners. For example, if you've got a company that you often send commercially-sensitive information to (health care providers and insurance companies are good examples), you can use a Message Transfer Agent (MTA) that actually does make sure that the digital certificates match and that you're talking to who you think you are. Even if you don't have that kind of requirement, though, enabling TLS will improve your overall e-mail security.

Next page: Move to S/MIME Next >>

This was last published in March 2005

Dig Deeper on Email and Messaging Threats-Information Security Threats