Because a security association is not established with S/MIME, what happens if the receiver does not support the message digest algorithm used by the sender?
A security association is a relationship between two or more entities that describes how they will utilize security services to communicate securely. For example, a security association is established in an SSL session during the handshake. When a client wishes to establish an SSL connection, it sends a CLIENT-HELLO message that includes the cryptographic systems it is willing or able to support. The server responds with a SERVER-HELLO message that also includes information about the cryptosystems it supports. The client is then responsible for choosing the cryptosystem it will share with the server to establish a security association. When you digitally sign an e-mail message, you do not necessarily know what mail client the recipient uses and the cryptosystems it supports, therefore a security association cannot be established. As your question implies, this could lead to a situation where the recipient's mail client doesn't support the algorithm used to create the message digest or hash value.
Normally, when a digitally signed message is received, the e-mail client calculates its own hash of the message and uses the sender's public key to decipher or decrypt the hash value, which is included with the message, to compare the two values. If the two values match, the recipient of the signed message can be sure it has not been altered and was signed by the owner of the private key. If the recipient's e-mail client cannot calculate the message digest because it doesn't support the sender's algorithm, it won't be able to verify either the sender's authenticity, or the integrity of the message.
The S/MIME protocol addresses this problem. Version 3 of S/MIME, which is supported by most popular e-mail clients, consists of several RFCs. One of these, RFC 3370, identifies the algorithms that S/MIME's version 3 software must support. These include Secure Hash Algorithm 1 (SHA-1) and Message Digest-5 (MD5) for hashing; Digital Signature Algorithm (DSA) and RSA for signatures; and RC2 and triple Data Encryption Standard (3DES) for message encryption. This ensures a base level of interoperability among all S/MIME implementations, but e-mail clients can add additional algorithms if they correctly identify which algorithms a particular message uses. The actual message digest algorithm used to sign a message, is stored in various fields in the SignedData value including the SignerInfo digestAlgorithms field. According to the RFC, if an e-mail client doesn't recognize or support this algorithm, it must "recover gracefully," which means the user will be notified that the client cannot correctly verify the message.
However, if the sender already obtained the recipient's pubic key and digital certificate from a previous message, X.509 digital certificates would use signature algorithm identifiers so the sender's e-mail client can use that information to select an appropriate algorithm. This means a security association can be established prior to the transmission and receipt of the message. Finally, remember that the cryptographic functions used during the exchange, can also be established based on out-of-band information including private agreements, user preferences and legal restrictions.
This was first published in February 2006