I am assuming you're trying to look at packets from an outside source that has several possible encryption methods,...
and you want to point the packet to the right decryption tool. The first answer is "yes," the second answer is "this may be an illegal activity." In many countries it's illegal to reverse-engineer protection mechanisms like encryption.
With that warning, if you wish to proceed, looking at the binaries (DLLs, EXEs, etc.) in a hex editor may show strings that indicate a specific cryptographic hash algorithm. You may also find strings that indicate a specific third-party encryption library.
Also, check the names of the DLLs. If, for example, ssleay.dll or libeay.dll is present, then it's easy to figure out that the packet is encrypted with SSL. If the encryption uses a third-party library, look up the functions exported by that library and see what parameters they take and how they are used. You can then trap the calls you're interested in: For example, with LIBeay or SSLeay, you'd be looking at ssl_read and ssl_write. This will give you access to the plaintext, and you can then start dumping whole session and examining the raw protocol.
If the encryption appears to be built into the executable, or the encryption writer appears to be using his or her own code, then you're going to have to start poking around with a debugger, following where data goes after socket reads. This should help you locate the decryption routines. Keep in mind that these activities will require pretty extensive experience with debugging tools and executable editors, so if you're not familiar with these, then my final answer would be "no."
For more information:
- Read more about public key cryptography algorithms.
- Check out this video in which Bruce Schneier discusses the state of cryptography.
Dig Deeper on Disk and file encryption tools
Related Q&A from Randall Gamby
Learn how to create account lockout policies that detail how many unsuccessful login attempts are allowed before a password lockout in order to ... Continue Reading
When it comes to minimum password length, 14-character passwords are generally considered secure, but they may not be enough to keep your enterprise ... Continue Reading
Enterprise SSO products have matured over the years, so what's the state of eSSO today? Expert Randall Gamby discusses. Continue Reading