Problem solve Get help with specific problems with your technologies, process and projects.

Detecting malware encryption: Can NGFWs spot SSL-encrypted malware?

An NSS Labs study claims next-generation firewalls struggle to detect and defend against SSL-encrypted malware. Brad Casey analyzes the findings.

A study by NSS Labs found that many NGFWs have trouble dealing with SSL-encrypted traffic, with malware using such measures being difficult to detect. Are there any alternative measures that enterprises can take to defend against SSL-encrypted malware? Or can NGFWs be tuned in such a way that increases detection?

Ask the Expert

Want to ask Brad Casey a question about network security? Submit your questions now via email. (All questions are anonymous.)

According to the NSS Labs report, after enterprise-grade levels of SSL traffic were thrown at eight different next-generation firewalls (NGFWs), an average performance loss of 81% was observed. The report states that payloads utilizing 2,048-bit encryption ciphers were leveraged during testing, and while an extremely high traffic volume was used in the simulation, testers cautioned that organizations should expect the volume of enterprise SSL traffic to increase approximately 20% annually due to exponential growth in smartphone and tablet use.

It becomes readily apparent the further one reads through the report that the handling of SSL-encrypted traffic is a hindrance on network performance, and what becomes even more apparent is that, according to different sources cited within the report, only approximately 1% of existing malware utilizes SSL encryption. So the question then becomes, is examining all SSL traffic that enters and exits the network worth the loss in performance?

Fortunately, this doesn't have to be an either-or proposition -- at least not completely. To defend against SSL-encrypted malware, an enterprise could implement the offload method utilized by Sourcefire Inc. and cited within the report. In this arrangement, a traditional firewall is deployed for non-SSL traffic, and a separate but correlating appliance (likely a NGFW) is deployed to examine all SSL traffic. Per the report, this configuration resulted in minimal performance impact for the Sourcefire 8250 and 8290, the devices that performed best out of all the firewalls tested.

Alternately, enterprises could take a host-based approach toward SSL-encrypted malware defense. In this implementation, all SSL traffic is allowed through the firewall, and the burden of examining it is placed on each individual node. This approach may work well when certain pockets of a network access more SSL traffic than others -- for example, financial departments using encryption when communicating with financial institutions and vice versa. Utilizing this approach keeps one portion of the network (the financial departments in this case) from cannibalizing network resources at the expense of non-SSL users.

Lastly, one could simply stick with the status quo. If a risk analysis is done in conjunction with what the expected performance loss would be, it may find it more feasible to accept the risk malware encryption may pose.

This was last published in December 2013

Dig Deeper on Network device security: Appliances, firewalls and switches

Have a question for an expert?

Please add a title for your question

Get answers from a TechTarget expert on whatever's puzzling you.

You will be able to add details on the next page.

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.