Google's Project Zero discovered a serious Cloudflare bug that exposed private customer data for potentially millions...
of websites hosted on Cloudflare's content delivery network.
The exposure is the result of a memory leak caused by a buggy interaction between an older HTML parser and one Cloudflare began using last September. Cloudflare Inc., which mitigated the bug, stated in its incident report Thursday that only about one in 3.3 million HTTPS requests could have exposed sensitive customer information, including personally identifiable information (PII), passwords, cookies and tokens.
However, the Cloudflare bug, which was first discovered by Google Project Zero researcher Tavis Ormandy, made sensitive information available to anyone running a web crawler on Cloudflare CDN customers' sites since last September. As a result, Cloudflare customer data transmitted between Sept. 22, 2016, and Feb. 17, could have been exposed to third parties. It's unclear how many of Cloudflare's customers and hosted websites were actually exposed, as the company is still investigating the incident and notifying customers.
"This situation was unusual," Ormandy wrote in the issue tracker record. "PII was actively being downloaded by crawlers and users during normal usage; they just didn't understand what they were seeing."
Ormandy found "private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites [and] hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything."
Cloudflare responded by identifying three CDN features that were using the HTML parser in question, and then turned them off. "The bug was serious because the leaked memory could contain private information and because it had been cached by search engines," stated John Graham-Cumming, a Cloudflare programmer, in Cloudflare's incident report. "We have also not discovered any evidence of malicious exploits of the bug or other reports of its existence."
Graham-Cumming wrote that "some" customer data was cached by search engines, and Cloudflare worked with Google and other search engine companies to remove that cached data. The Project Zero team shared redacted screenshots of PII related to users of Uber, OkCupid and Fitbit; Cloudflare has more than 5 million websites in its content delivery network, and anyone who uses Cloudflare-hosted websites could be affected. Sven Slootweg, an independent security researcher, tweeted:
So in case this wasn't clear yet:— Sven Slootweg (@joepie91) February 23, 2017
Consider ALL sensitive data EVER sent to ANY site on CloudFlare to be irrevocably compromised. Everything.
Ormandy and others praised Cloudflare's quick response. "After I explained the situation, Cloudflare quickly reproduced the problem, told me they had convened an incident and had an initial mitigation in place within an hour," Ormandy wrote.
Rob Graham, owner of Errata Security, tweeted:
This is how you know you can trust CloudFlare: yes, they have bugs, but they don't run and hide from them. https://t.co/pJ1JsLolZh— Rob Graham (@ErrataRob) February 23, 2017
Graham-Cumming said "the greatest period of impact" for the Cloudflare bug was between Feb. 13, and Feb. 18, which just happened to be during RSA Conference 2017 -- one of the busiest times of year for the security industry.
Cloudflare said in the course of rolling out an improved program to parse HTML data, it had inadvertently caused a conflict with its older parser; the result was that instead of properly terminating while reading data out of memory, the software retrieved -- and delivered to browsers, robots and web scrapers -- data from uninitialized memory.
"Unfortunately, it was the ancient piece of software that contained a latent security problem, and that problem only showed up as we were in the process of migrating away from it," Graham-Cumming wrote. "Our internal infosec team is now undertaking a project to fuzz older software looking for potential other security problems."
Find out more about improving content delivery network security
Learn about the importance of HR programs to secure personally identifiable information
Read about some of the free tools available to uncover risks to personally identifiable information