News Stay informed about the latest enterprise technology news and product updates.

Remote access Trojan evades detection using mouse functions

Trojan highlights need to tune automated detection systems to spot malicious software attempting to use Windows hooks, expert says.

Malware developers are continuing to deploy clever agents that are designed to avoid automated detection by hiding...

behind operating system routines, but the same mechanisms these types of malware employ to stay stealthy could have been used to flag the code as malicious all along.

The antivirus industry is so overwhelmed and short of analysts these days, they don’t have time to manually analyze such code.

Kevin McAleavey, co-founder and architect, KNOS Project

Researchers at Symantec recently identified a remote access Trojan (RAT) that is capable of evading detection by hiding behind routines that are used to communicate with external devices such as a mouse, evading detection because the code remains inactive when the mouse is not in use, as is the case during examination by dynamic analysis systems in a virtual environment.

"If malware can hide itself from automated threat analysis systems, it can blend in with millions of sample files and antivirus applications may not be able to figure out that it is malicious. Therefore, both malware and packer program authors attempt to utilize techniques to hide malicious files from automated threat analysis systems," wrote Symantec's Hiroshi Shinotsuka

Key to this type of malware’s ability to evade detection is its use of system-wide “hooks,” the message-handling mechanisms which allow applications such as a mouse driver to install a subroutine to monitor system message traffic for queues to activate, Shinotsuka explained. The malware Symantec identified exploits a “wait hook” in order to remain dormant until it is activated by the use of the mouse.

The problem is that the exploitation of various system “hooks” by malware developers has been occurring for years, and dynamic analysis systems should have long ago been designed to compensate for this blind spot in threat detection, according to Kevin McAleavey, co-founder and architect of the KNOS Project.

"System-wide hooks as indicated in the Symantec discovery are old hat and should have been checked for all along by automated analysis, as well as any attempts by a legitimate program to engage hooks of any kind," said McAleavey who has been active in anti-malware research and security product development since 1996.

The issue comes down to a matter of manpower. Given that there is an average of one million new malware variants identified each day, it is impossible to manually examine the function of each sample in order to assess which warrants the immediate creation of a definition for inclusion in antivirus software, so it is necessary to automate threat analysis with systems that are used to analyze code behavior in a more efficient manner.

Since automated detection may fail to identify malware such as the RAT Symantec discovered, McAleavey believes that any application that uses a “hook” should be flagged for manual analysis, but given the shortage of personnel with the prerequisite expertise for malware analysis, this is unlikely to occur.

"Functions employing a hook of any type should be reason enough to trigger a very careful manual analysis. But then again, the antivirus industry is so overwhelmed and short of analysts these days, they don’t have time to manually analyze such code," McAleavey says.

A detailed listing of the various “hookable” Windows events is publicly available in Microsoft's MSDN documentation. As an example, McAleavey points out the various things that a “wait hook” can be set on, any of which could be used by malware designers to activate the code:

  •     WH_CALLWNDPROC - wait for a specific message passed from one window to another
  •     WH_CBT - an oldie which supported "computer based training" and has been used for keystroke record/playback macros
  •     WH_GETMESSAGE - retrieval of mouse, keyboard or any other system input
  •     WH_KEYBOARD - keyboard hook, the preferred attack vector for keyloggers
  •     WH_MOUSE - mouse hook, which is what Symantec is talking about
  •     WH_MSGFILTER - detects touching of a button, menu or other Windows feature
  •     WH_DEBUG - any overflow which would trigger a debugger

McAleavey explains that the “hook” functions listed above are only a few of the “debug” functions available for exploitation, such as those that are able to detect if a machine is running in a virtual mode, those that check to see if an antivirus has inserted its own hooks for detection purposes, and those that divert programs into separate malicious DLL's in order to control whether or not an infection is detected at all.

“The WH_DEBUG hook controls all access to the other WH hooks and can interrupt any of the other hooks in order to check for example whether or not the hooks should proceed or to replace the original functions with some other action. This would be how malware could detect if an antivirus is monitoring the other hooks, and if so, prevent detection,” McAleavey said.

The problem McAleavey has with Microsoft's debug functions is that they're present in the retail versions of the operating system itself. Microsoft already provides developer versions of all of their operating systems which contain voluminous debug functions and debug symbols for use in writing software for the Windows platform, so their presence in the retail versions is unnecessary.

"Bottom line, Microsoft's debug functions have always been used by malware in order to hide themselves. The first thing human analysts will do is review the function calls of any sample to detect the presence of any debug calls and then follow them carefully. I'm quite surprised that no flags have been raised over yet another debug call by automated systems,” he said.

McAleavey says that since each of the debug functions is enumerated and listed in all executables, any automated system should be able to easily spot their existence in a submitted sample and act on them. Any call to a debug hook in a program will leave the call visible in the program itself, so any automated analysis system could potentially spot the call to SetWindowsHookEx and flag it for analysis. These calls are not hidden; they are present in the sample being analyzed, so all the automated systems need to do is match the characters when scanning a file.

“It seems counter-intuitive to have ignored them all this time. Legitimate software should not be making debug calls such as this without a very specific reason to do so, and unknown executables should be immediately flagged when any of these functions are detected and remain flagged until proven inert,” McAleavey said.

About the author:
Anthony M. Freed is an information security journalist and editor who has authored numerous feature articles, interviews and investigative reports which have been sourced and cited by dozens of major media outlets. You can also find him tweeting about security topics on Twitter @anthonymfreed.

Dig Deeper on Malware, virus, Trojan and spyware protection and removal

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.