The real answer is that both are used. IPsec and SSL use asymmetric encryption to establish the encryption protocol when the session starts and then to securely exchange a private key used during the session. This private key is similar to the single secret key used in symmetric encryption.
Asymmetric encryption uses a key pair -- both a public and a private one -- for encryption. The sender uses the receiver's public key to encrypt the data and the receiver uses their private key to decrypt it. The transmission is secure because the recipient always has the private key in their possession and never exposes it by sending it over a public connection, such as the Internet. On the other hand, the public key, which is openly exposed in transit over the wire, cannot derive the private key. The two keys are only mathematically related and nothing more. So, even if sniffed en route, the public key is useless by itself.
Symmetric encryption uses only a single secret key by itself. However, since IPsec and SSL by nature communicate openly across the Internet, a captured secret key would defeat symmetric encryption. A malicious user, using the encryption algorithm, if known, could then use the key to decrypt any traffic transmitted over the wire between the two hosts.
To clarify some terminology here, symmetric encryption uses what's called a secret key. This isn't meant to be confused with the private key in asymmetric encryption, which like its symmetric counterpart, is also secret. The difference is that the secret key in symmetric encryption is a single key, while the private key in asymmetric encryption is part of a key pair.
However, there is a catch to using asymmetric encryption. It runs about 1,000 times slower than symmetric encryption and eats up just as much processing power, straining already overburdened servers. That means asymmetric encryption is only used (by IPsec and SSL) to create an initial and secure encrypted connection to exchange a private key. Then, that key is used to create a shared secret, or session key, that is only good during the session when the two hosts are connected.
This was first published in November 2005