Problem solve Get help with specific problems with your technologies, process and projects.

Antivirus engines: Lessons learned from the Tavis Ormandy Sophos research

Learn how the discovery of several flaws in the Sophos antivirus engine can help advance the state of antimalware software.

Since its inception, the antivirus industry has operated under the belief that it should function in obscurity....

The theory, in a nutshell, was always that if only a small group of people know exactly how antivirus engines detect malware, that would mean malicious hackers would be less likely to have the knowledge necessary to subvert antimalware systems. As a result, most of the AV industry’s operations have been closely guarded secrets; that is, until recently, thanks to antivirus research by Tavis Ormandy.

Ormandy, an information security engineer at Google Inc. and a well-known bug hunter, took a deep look at the Sophos AV engine and found some fundamental flaws with the technology itself, which he published in a white paper and presented at Black Hat 2011. In this tip, we examine what Ormandy discovered, why his findings are important to the success of antimalware, the validity of his assertion that commercial AV engines are inherently flawed and, finally, enterprise protection strategies.

Few vendors have totally rewritten the core functionality of their AV engines recently, and most have failed to update their code to use modern software development practices.

Security flaws in Sophos antivirus engine
Ormandy identified several high-level flaws in the Sophos antivirus engine related to core signature functionality, buffer overflow protections and the use of cryptography. Ormandy found the AV signature quality to be relatively poor, and that attackers could easily manipulate the signatures. The buffer overflow protection is reported to only work on Windows Vista and earlier versions of Windows, contrary to Sophos’ published support. It can also be bypassed by an attacker. The general use of weak cryptographic algorithms underlies many different areas, and the encryption key is embedded in the product.  While some question his motives in singling out Sophos, Ormandy’s research will help improve defenses and causes enterprises to carefully evaluate if their antivirus engines may be vulnerable to similar attacks. By broadly publishing his research, the entire industry is able to benefit from his research.

Ormandy’s findings are important to the success of current antimalware engines. Few vendors have totally rewritten the core functionality of their AV engines recently, and most have failed to update their code to use modern software development practices. This concern is not unique to Sophos or antivirus engines. Not updating old code is common throughout the software industry.

Ormandy’s assertion that commercial antivirus engines are inherently flawed, however, is not news; it is something the information security industry has been grappling with for the last few years. Many enterprises rely heavily on commericial antivirus engines in products like those from Sophos, Symantec, McAfee and other vendors to protect their systems from a broad range of attacks. Many experts pan antivirus as yesterday's technology, but it's still an important technology that should be used with other security controls, like timely patch deployments.

However, Ormandy's research should serve as a wake-up call for enterprises that rely solely on antivirus systems without combining the technology with other security controls. Few enterprises have the resources to do the in-depth research Ormandy performed to ensure they are aware of the specific challenges with their chosen solutions. Even independent antivirus evaluations from Virus Bulletin, AV-Comparatives, West Coast Labs and other third parties had not discovered the issues with Sophos’ AV engine because they focus on detection performance. So it's likely the antivirus engines from other commercial vendors suffer from other similar problems that simply haven't been discovered yet.   

Enterprise malware protection strategies
The flaws Ormandy outlined are serious and should prompt enterprises to carefully evaluate if Sophos, or any antivirus vendor, is meeting their needs, but enterprises should already be carefully evaluating their security controls for effectiveness. Sophos responded to Ormandy’s findings in a blog post and recommends that enterprises keep their systems patched and running current versions of the antivirus engine.

Enterprises may also consider using different antivirus engines on servers or network devices than those on desktops. This may provide some additional protection since other engines most likely do not have the same flaws, unless parts of the engine are licensed from Sophos. This new engine will add to the complexity of the environment, so using a different antivirus engine should be very carefully weighed against other security controls that could prevent malware like application white listing, adding a host intrusion prevention system, or even using a supplemental antimalware engine on the endpoint.

The research Ormandy performed will help advance the state of antimalware software. It will hopefully prompt Sophos and other antivirus vendors to carefully evaluate their core software to ensure the protections work as expected. The issues Ormandy identified are likely not unique to Sophos, as other vendors' antivirus engines are architected similarly. Enterprises should learn from the Tavis Ormandy Sophosincident and not only press their antimalware vendors on how they're advancing their core product offerings to stay ahead of attackers, but also to make sure their antivirus programs are supplemented with some of the other technologies noted above that can prevent malware and help achieve an effective defense-in-depth architecture.

About the author:
Nick Lewis, CISSP, is an information security architect at Saint Louis University. Nick received his Master of Science in Information Assurance from Norwich University in 2005 and Telecommunications from Michigan State University in 2002. Prior to joining Saint Louis University in 2011, Nick worked at the University of Michigan and at Children's Hospital Boston, the primary pediatric teaching hospital of Harvard Medical School, as well as for Internet2 and Michigan State University.

This was last published in November 2011

Dig Deeper on Endpoint protection and client security

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.