I am assuming you're trying to look at packets from an outside source that has several possible encryption methods, and you want to point the packet to the right decryption tool. The first answer is "yes," the second answer is "this may be an illegal activity." In many countries it's illegal to reverse-engineer protection mechanisms like encryption.
With that warning, if you wish to proceed, looking at the binaries (DLLs, EXEs, etc.) in a hex editor may show strings that indicate a specific cryptographic hash algorithm. You may also find strings that indicate a specific third-party encryption library.
Also, check the names of the DLLs. If, for example, ssleay.dll or libeay.dll is present, then it's easy to figure out that the packet is encrypted with SSL. If the encryption uses a third-party library, look up the functions exported by that library and see what parameters they take and how they are used. You can then trap the calls you're interested in: For example, with LIBeay or SSLeay, you'd be looking at ssl_read and ssl_write. This will give you access to the plaintext, and you can then start dumping whole session and examining the raw protocol.
If the encryption appears to be built into the executable, or the encryption writer appears to be using his or her own code, then you're going to have to start poking around with a debugger, following where data goes after socket reads. This should help you locate the decryption routines. Keep in mind that these activities will require pretty extensive experience with debugging tools and executable editors, so if you're not familiar with these, then my final answer would be "no."
For more information:
- Read more about public key cryptography algorithms.
- Check out this video in which Bruce Schneier discusses the state of cryptography.
Dig Deeper on Disk Encryption and File Encryption
Related Q&A from Randall Gamby, Contributor
Is your remote desktop access software really secure? Randall Gamby offers advice for conducting a remote access audit to validate security.continue reading
Expert Randall Gamby discusses risk-based authentication, and whether that type of user identification system is right for the enterprise.continue reading
Expert Randall Gamby discusses various types of single sign-on, specifically the approaches of Ping Identity's SSO and Symplified SSO.continue reading
Have a question for an expert?
Please add a title for your question
Get answers from a TechTarget expert on whatever's puzzling you.