I am assuming you're trying to look at packets from an outside source that has several possible encryption methods,...
By submitting your email address, you agree to receive emails regarding relevant topic offers from TechTarget and its partners. You can withdraw your consent at any time. Contact TechTarget at 275 Grove Street, Newton, MA.
and you want to point the packet to the right decryption tool. The first answer is "yes," the second answer is "this may be an illegal activity." In many countries it's illegal to reverse-engineer protection mechanisms like encryption.
With that warning, if you wish to proceed, looking at the binaries (DLLs, EXEs, etc.) in a hex editor may show strings that indicate a specific cryptographic hash algorithm. You may also find strings that indicate a specific third-party encryption library.
Also, check the names of the DLLs. If, for example, ssleay.dll or libeay.dll is present, then it's easy to figure out that the packet is encrypted with SSL. If the encryption uses a third-party library, look up the functions exported by that library and see what parameters they take and how they are used. You can then trap the calls you're interested in: For example, with LIBeay or SSLeay, you'd be looking at ssl_read and ssl_write. This will give you access to the plaintext, and you can then start dumping whole session and examining the raw protocol.
If the encryption appears to be built into the executable, or the encryption writer appears to be using his or her own code, then you're going to have to start poking around with a debugger, following where data goes after socket reads. This should help you locate the decryption routines. Keep in mind that these activities will require pretty extensive experience with debugging tools and executable editors, so if you're not familiar with these, then my final answer would be "no."
For more information:
- Read more about public key cryptography algorithms.
- Check out this video in which Bruce Schneier discusses the state of cryptography.
Related Q&A from Randall Gamby
Which authentication method is better for securing enterprise devices and systems: two-factor authentication or multifactor authentication?continue reading
Securing biometric information is a crucial step for enterprises to take, but what happens if the data is still compromised? Expert Randall Gamby ...continue reading
Simple photography cracking biometric systems highlights the need for two-factor authentication in enterprises according to expert Randall Gamby.continue reading
Have a question for an expert?
Please add a title for your question
Get answers from a TechTarget expert on whatever's puzzling you.