I am assuming you're trying to look at packets from an outside source that has several possible encryption methods, and you want to point the packet to the right decryption tool. The first answer is "yes," the second answer is "this may be an illegal activity." In many countries it's illegal to reverse-engineer protection mechanisms like encryption.
With that warning, if you wish to proceed, looking at the binaries (DLLs, EXEs, etc.) in a hex editor may show strings that indicate a specific cryptographic hash algorithm. You may also find strings that indicate a specific third-party encryption library.
Also, check the names of the DLLs. If, for example, ssleay.dll or libeay.dll is present, then it's easy to figure out that the packet is encrypted with SSL. If the encryption uses a third-party library, look up the functions exported by that library and see what parameters they take and how they are used. You can then trap the calls you're interested in: For example, with LIBeay or SSLeay, you'd be looking at ssl_read and ssl_write. This will give you access to the plaintext, and you can then start dumping whole session and examining the raw protocol.
If the encryption appears to be built into the executable, or the encryption writer appears to be using his or her own code, then you're going to have to start poking around with a debugger, following where data goes after socket reads. This should help you locate the decryption routines. Keep in mind that these activities will require pretty extensive experience with debugging tools and executable editors, so if you're not familiar with these, then my final answer would be "no."
For more information:
- Read more about public key cryptography algorithms.
- Check out this video in which Bruce Schneier discusses the state of cryptography.
This was first published in March 2010