Ask the Expert

How to decode a cipher: Identifying a cryptographic hash algorithm

Is it possible to identify a cryptographic algorithm from cipher bit sequences?

    Requires Free Membership to View

I am assuming you're trying to look at packets from an outside source that has several possible encryption methods, and you want to point the packet to the right decryption tool. The first answer is "yes," the second answer is "this may be an illegal activity." In many countries it's illegal to reverse-engineer protection mechanisms like encryption.

With that warning, if you wish to proceed, looking at the binaries (DLLs, EXEs, etc.) in a hex editor may show strings that indicate a specific cryptographic hash algorithm. You may also find strings that indicate a specific third-party encryption library.

Also, check the names of the DLLs. If, for example, ssleay.dll or libeay.dll is present, then it's easy to figure out that the packet is encrypted with SSL. If the encryption uses a third-party library, look up the functions exported by that library and see what parameters they take and how they are used. You can then trap the calls you're interested in: For example, with LIBeay or SSLeay, you'd be looking at ssl_read and ssl_write. This will give you access to the plaintext, and you can then start dumping whole session and examining the raw protocol.

If the encryption appears to be built into the executable, or the encryption writer appears to be using his or her own code, then you're going to have to start poking around with a debugger, following where data goes after socket reads. This should help you locate the decryption routines. Keep in mind that these activities will require pretty extensive experience with debugging tools and executable editors, so if you're not familiar with these, then my final answer would be "no."

For more information:

This was first published in March 2010

There are Comments. Add yours.

TIP: Want to include a code block in your comment? Use <pre> or <code> tags around the desired text. Ex: <code>insert code</code>

REGISTER or login:

Forgot Password?
By submitting you agree to receive email from TechTarget and its partners. If you reside outside of the United States, you consent to having your personal data transferred to and processed in the United States. Privacy
Sort by: OldestNewest

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: