Antivirus software comparison, 2004: Not all AV products are equal

Your desktop AV may be leaving you wide open to attack.

This article can also be found in the Premium Editorial Download: Information Security magazine: Exposed: Why your AV software is failing to protect you:

Your antivirus product isn't doing the job you think it is.

AV is one of the most mature infosecurity technologies and the undisputed mainstay of desktop and network defense against nasty code lurking on the Internet. But, too many of us put unwavering trust in these applications to stop malware attacks.

Such blind faith is misplaced, as Information Security found in its month-long test of 10 leading desktop AV products against 11 criteria.

AV has barely made a dent in detecting spyware or backdoors.



We found that many AV solutions are surprisingly easy to defeat, can't detect malware using alternative attack vectors and are difficult to manage. Strikingly, the capabilities and reliability of the products varied greatly.

In the two decades since the first viruses appeared, most AV vendors continue to push the same basic signature-based technology. Feature sets have been added and functionality improved, but the products haven't evolved as rapidly as the capabilities of viruses and worms.

For this reason, rather than simply testing the breadth and effectiveness of vendors' signature libraries, we focused on other critical--and often neglected--aspects of AV products: effectiveness against attack mechanisms designed to fool or disable AV protection; detection of increasingly popular forms of malware such as spyware and backdoors; and, in particular, enterprise-scale manageability (see "Report Card").

We discovered that not all AV products are equal, and many don't provide the protection you think they do.

Evading AV
Malware writers are constantly looking to give their malcode the ability to evade detection and gain unfettered control over target systems. Signatures won't work if malware can dodge AV protection. To test the AV tools' resilience, we exposed each product to malware for which it had a signature, then attacked it using various techniques to fool or disable it. We found that some products were easily subverted, while others were made of sterner stuff.

Delving Into Hidden Streams
Attackers can--and often do--insert malware into an alternate data stream (ADS) under a directory or file on a Windows NTFS partition. ADSes are subterranean structures hiding inside the file system and are invisible to built-in Windows directory tools. Attackers can easily tuck malicious code into an ADS using a variety of Windows programs, including a simple "type" command.

There was something of a hubbub in 2000 because most AV products didn't scan ADSes. Most vendors say they've closed this gap, but the problem hasn't gone away. Not only were we able to plant malware in ADSes under executables, text files and directories, but the majority of the tested AV products ignored ADSes during on-demand scans when set to default configurations. Even more chilling, several products overlooked ADS-borne malware in real-time scans.

Only Network Associates detected malware in ADSes during both on-demand and real-time scans with its default configuration, which offers a solid level of security.

Default real-time protection against ADS-borne malware is also provided by Computer Associates (CA), F-Secure, Grisoft, Panda Software and Sophos.

Kaspersky Labs and PestPatrol, on the other hand, offer no real-time protection against malware in ADSes. In these products, an on-demand scan will detect ADS-borne malware--unless a user disables it.

Symantec and Trend Micro offer protection, provided it's turned on. Symantec's default configuration provides no malware protection in ADSes--a significant concern. Trend Micro's default settings check for ADS-based malware attached to .exe files but completely ignore code stashed under other file types. These defaults are dangerous because the vast majority of organizations don't tune their desktop AV settings.

Resisting AV Killers
Many malware specimens, including last summer's Bugbear.B worm, attempt to disable AV before infecting a target system. Here's how it works: A worm initially launches a small "warhead" to disable AV by killing its processes, then installs a backdoor. Alternatively, a clueless or malicious user might try to disable a scan or kill its running processes.

We attempted to kill the processes associated with each AV product to disable real-time protection by running a kill script that simulates the action of malware. The good news is that malicious code or users without admin rights couldn't shut down any of the AV solutions. But the results were mixed when we tried the same thing using malware with admin privileges.

This is a serious concern: There's the threat of malicious code running with admin or system privileges, and many organizations give users local admin privileges on their desktops--a dangerous and frustratingly persistent practice.

Only Kaspersky was bulletproof against this attack. We couldn't shut down its AV processes, even as admin-level users. Kaspersky kept on running, displaying an error message as it thwarted our attempts. Some of the others keeled over and died, while the balance fell somewhere in between:

  • As a local admin, we disabled real-time AV protection on systems running CA, F-Secure and PestPatrol, rendering them vulnerable to malware until the next on-demand scan or reboot.
  • We temporarily disabled Panda's and Trend Micro's AV protection, but the processes restarted automatically within 30 seconds. While that's certainly better than no protection, it leaves a brief window of exposure.
  • We were able to kill AV processes on Network Associates- and Sophos-protected machines, but doing so crashed the systems and forced a reboot. It wasn't graceful, but the machines weren't compromised.
  • We could kill processes associated with Grisoft and Symantec, but AV protection never ceased.

Compressing and Concealing
Attackers also attempt to hide malware by compressing it. We evaluated the AV products' ability to detect common malware specimens compressed with WinZip, Win-Zip twice compressed, Tar, gzip and Bzip2 formats.

F-Secure, Kaspersky, Network Associates and Panda rose above the rest by detecting compressed malware in several formats. However, the default configurations of CA, PestPatrol and Sophos couldn't detect tar-compressed malware, which can easily be opened with WinZip and is popular for distributing malicious code.

Spotting Modified Malware
For most of the tests in this review, we used unmodified malware specimens that were safely downloaded and moved to our air-gapped lab. But we also wanted to see how AV products would respond to a very small and common tweak of some popular backdoor tools--tini.exe, Ultor's Trojan Port and NTbindshell.exe--each of which provides a backdoor shell that listens on a TCP port.

Most AV vendors focus on detecting malware files that are exactly the same as those found in the wild. Using a hex editor, we made slight alterations, modifying just two bytes in the executable itself so that the program would listen on a different TCP port. This is a common tactic among attackers, who can customize ports in a backdoor based on what's accessible in the target network.

Only Network Associates, Panda and Trend Micro detected all of our modified malware specimens.

Beyond Viruses and Worms
In addition to viruses and worms, attackers plant hidden backdoors to control machines and use Windows boxes to stage attacks into Unix and Linux systems. Aggressive Web sites and intruders push spyware onto users' machines to log keystrokes and track surfing habits.

We see this stuff as we respond to computer incidents, but can AV solutions protect us? As it turns out, no, most of them can't.

The tested AV products focus almost exclusively on detecting self-replicating malicious code (viruses and worms) but almost never detected spyware or backdoors. Overall, the results were rather disappointing.

Finding Backdoors
Attackers install backdoors and trick users into running them to gain remote access to the heart of your enterprise. The most popular programs used to create backdoors are Netcat, which creates a remote command-shell listener; and the Virtual Network Computing (VNC) suite, used to control a machine's GUI across a network. Because of their immense usefulness, both security pros and black hats use these tools in their protection/penetration regimen. Even though they have legitimate uses, most security personnel want a heads up when they get installed--potentially by an attacker.

Eight of the 10 AV tools left our test systems wide open to Netcat and VNC. Only PestPatrol detected both; Panda detected VNC.

Rooting Out *nix-based Malware
AV vendors focus their efforts on most malware writers' target of choice: Windows. However, attackers can also use compromised Windows desktops as a staging ground to put rootkits and other malware onto Unix and Linux boxes.

While Windows-based AV products can provide early warnings to detect files that could be used to attack *nix systems, by and large, they don't.

The results were disappointing when we exposed Windows-based AV products to the most popular malware for Unix and Linux, including the Linux Rootkit 5 (LRK5), the Universal RootKit (URK), Adore, Kernel Intrusion System (KIS) and the SuperUser Control Kit. Kaspersky, Network Associates, Panda, Symantec and Trend Micro detected only LRK5, while the other products ignored all of the *nix malware.

Unmasking Spyware
Lately, the Web seems like a cesspool of aggressive spyware. Keystroke-loggers running on a manager's desktop can learn user IDs and passwords or reveal sensitive business information.

It's alarming, but don't count on AV products for protection.

We exposed the AV products to 15 common spyware programs, including SaveKeys (a commercial keystroke logger), Perfect Keylog-ger Lite (a free version of a popular keystroke logger), FreeScratch- AndWin (a tool that inserts ads into users' browsing experience) and the controversial Gator software (which aggregates user surfing habits for the commercial benefit of advertisers).

PestPatrol, which specializes in detecting malware other than worms and viruses, performed relatively well, identifying 10 of the 15 specimens, and was the only application to detect Gator. F-Secure, Kaspersky, Network Associates and Panda all detected the same three or four spyware programs. CA, Grisoft, Sophos and Trend Micro detected either one or none of our spyware samples.

For their part, most of the AV vendors maintain that these spyware programs are simply doing what they advertise: recording information about users' actions, consistent with their README files and EULAs. The vendors won't classify these tools as malware, regardless of their impact on users and enterprises. Some vendors even cite legal concerns to characterize commercial keystroke loggers as malicious code.

Managing AV
No matter how adept desktop AV products are at detecting malware, they won't have much practical benefit for your organization without robust enterprise management capabilities, including remote configuration, status checking, data aggregation and analytical reports.

Reining in Users
Every tool we evaluated has options that prevent users from making arbitrary changes to AV configuration.

Grisoft, Network Associates, Trend Micro and Symantec offer solid, fine-grained configuration of the client interface, letting security managers control whether the user can alter dozens of different settings. For example, each of these tools allows admins to prevent users from disabling real-time scans or changing the signature up-date schedule.

Sophos, however, won't lock out users with local admin privileges. This is a serious concern because--as noted earlier--many organizations grant desktop users this level of control.

Reporting on the Enterprise
By aggregating data across the enterprise, good AV reports allow quick identification and mitigation of the biggest areas of risk. For example, reports summarizing infection history can show which users are regularly getting infected, which business units within the enterprise are keeping signatures up to date and which departments have the largest number of users who disable their AV products.

We found a mixed bag of reporting capabilities among the tested AV products. Network Associates and Trend Micro were the best, providing aggregate summaries of top infected hosts, descriptions of the most widely detected malware and detailed infection histories.

Both F-Secure and Sophos offered report options showing the most common infections, the most infected hosts and the signature update status for hosts. However, neither tool gave an at-a-glance summary of the infection or update status of the enterprise.

CA, Grisoft, Kaspersky, Panda, PestPatrol and Symantec provided minimal reports--essentially event logs from each client that needed to be manually aggregated or, in Symantec's case, parsed with a separately purchased tool.

Interface Useability
After installing each management tool and managed client, we judged the overall admin experience, including the determination of malware infestations across the enterprise, ease in updating signatures, simplicity of scheduling scans and alteration of managed client configurations on the fly.

Symantec and Trend Micro offered the most intuitive interfaces. Symantec provided an easy-to-use GUI for pushing installations to clients and instantly determining the status of the AV tool running on an individual host. Trend Micro's interface excelled in its ability to show the status of infections across the enterprise at a glance, with high-level alerts about particularly nasty specimens, and to drill down into detail with a simple point and click.

Network Associates' management GUI was solid, though not as intuitive as Symantec's or Trend Micro's. However, it was more complex than it probably needed to be.

The F-Secure and Grisoft interfaces were somewhat frustrating. With F-Secure, we had difficulty quickly determining the protection status on a given desktop. The "Alerts" tab in the management interface showed new attacks, but the "Status" tab showed no such indication, implying that the given host wasn't having any difficulties.

We found similar results with Grisoft, compounded on several occasions when the GUI locked out the admin for up to 30 seconds while checking the status of a disabled client--not a trivial point when managing mid- to large-sized enterprises. Additionally, the task scheduler was difficult to use.

Communicating on the Network
AV vendors use a wide variety of network protocols to communicate with managed clients, polling them for status, updating configuration settings and deploying new signature files. We sniffed each of these tools to determine the implications of allowing communication across an enterprise network.

We are leery of using Windows NetBIOS, SMB and Microsoft RPC networking for AV management because an increasing number of worms ride on those protocols, including last summer's Nachia/Welchia and Blaster. If your AV management server gets infected, it will rapidly spread the contagion to all of its managed clients.

Although most enterprises' internal networks allow wide-open Windows NetBIOS, SMB and Microsoft RPC access, many organizations are working to proactively filter and segment Windows network protocols on their internal networks. If a worm does sneak in, enterprises want to be able to apply filters on the fly to limit propagation.

CA, PestPatrol and Sophos all require Windows networking protocols for enterprise management functions, which open the door to many worms. If a worm spreads using these protocols, you can't filter it without affecting your ability to control your AV tool.

HTTP/HTTPS is a safer alternative. Admittedly, we've seen our fair share of HTTP-based worms, including Code Red and Slapper; they compromise unpatched Web servers to propagate. However, in most organizations, keeping Web servers patched is easier than keeping all of the Windows machines up to date, making HTTP/HTTPS a safer protocol for AV management. F-Secure, Network Associates and Trend Micro offer management capabilities using HTTP/HTTPS, along with a smattering of proprietary protocols.

Integrating with VPN for Enforcement
Some AV products offer enforcement capabilities to assure VPN users are running up-to-date AV software. Both Network Associates and Trend Micro interoperate with Check Point Software Technologies' Secure Configuration Verification (SCV) VPN solution, while Symantec works with Nortel and Check Point VPNs. When a user establishes a VPN connection, a corporate VPN gateway or local VPN software interrogates the AV client. If the AV tool is disabled or has an out-of-date signature base, the VPN won't let the client machine establish a connection to the corporate network. The client is forced to activate and/or update the AV tool before gaining access to the corporate network.

The Bottom Line
Our analysis makes it clear that AV products aren't the end-all, be-all defense against malicious code. As an industry, we haven't licked the ADS problem quite yet. AV has barely made a dent detecting spyware, backdoors and *nix malware, and enterprise-wide management and reporting capabilities vary significantly.

Trend Micro's solution, with its easy-to-understand GUI and HTTP networking, was the strongest overall based on our criteria.

It provided useful reports and fine-grained management, although its default ADS settings left something to be desired.

Network Associates came in a close second, besting Symantec in a head-to-head comparison of the AV giants. Network Associates' stellar ADS handling, combined with its solid management interface, comprehensive reports and fine-grained control of users, earned it high marks. Symantec offered a solid admin GUI, but its rating was hurt by weak built-in reporting capabilities and troubling default settings for identifying ADS-borne malware.

Panda's product was strongest among the small- and mid-sized vendors, providing solid detection capabilities and a decent management interface.

PestPatrol was an interesting product. The Corporate Edition's client has no user interface--it's completely transparent. If a user takes a laptop off of the network, PestPatrol will have no way of informing the user of what's happening, nor can the user invoke an on-demand scan. On the plus side, there's no way for a user to even think about changing settings. In addition to virus and worms, PestPatrol was adept at detecting malware overlooked by other solutions--some spyware and Netcat and VNC backdoors.

About the author:
Ed Skoudis, CISSP (, is cofounder of Intelguardians Network Intelligence, a security consulting firm, and author of Malware: Fighting Malicious Code (Prentice Hall, 2003).

Dig deeper on Security Awareness Training and Internal Threats



Enjoy the benefits of Pro+ membership, learn more and join.



Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to: