Heartbleed, the vulnerability in the open source OpenSSL encryption library, left organizations across the globe scrambling to apply patches in April. Security experts warned the flaw may expose enterprises' most sensitive of data, including keys used for X.509 certificates, user credentials and online communications.
For many organizations, including the University of Michigan, the Heartbleed problem was the ultimate incident response test. Though the university mustered a quick and effective response effort, the exercise raised questions regarding how to secure systems whose ownership was uncertain, as well as about how severe the bug ultimately was.
Heartbleed assessment: How bad was it?
Former University of Michigan chief security officer Paul Howell said the Ann Arbor-based higher education institution first learned about Heartbleed on April 7, when the school's Web team -- tasked with monitoring for patches related to the university's Web environment -- noticed an announcement on the OpenSSL website about the vulnerability. The Web team contacted Howell and his security staff about the notice, triggering the first step in the school's response: determining Heartbleed's severity.
There are issues and vulnerabilities all the time, but when one gets traction in the media the way Heartbleed did … then administrators start to hear about it.
former CSO, University of Michigan
Howell noted that in any given year, thousands of vulnerabilities will be made public across hundreds of products and that's excluding thousands more that are patched before security researchers are aware of them. Michigan strives to apply all relevant patches in a timely manner, he said, but as with all organizations, security updates must be prioritized based on the extent to which a vulnerability may affect a specific environment.
In the case of Heartbleed, Howell said that the Common Vulnerability Scoring System (CVSS) -- the NIST-run industry standard for measuring a vulnerability's severity -- rated the flaw as only a 5.0, or "medium." At that point, no known exploits existed publically, making an immediate attack that used Heartbleed unlikely.
Normally, such a vulnerability would not rank as a pressing priority, said Howell, but the potential damage that Heartbleed could cause -- unusually, the Heartbleed CVSS rating included a special note encouraging organizations to assess its impact in specific instances -- combined with an onslaught of media coverage meant the university had little choice but to act swiftly.
"There are issues and vulnerabilities all the time, but when one gets traction in the media the way Heartbleed did, and it's on the NBC Nightly News or NPR, then administrators start to hear about it," said Howell. "We had executive officers and others asking us what was going on."
In fact, in what is a reverse of most incident response scenarios, there was so much awareness of Heartbleed beyond the security team that Howell said he had to work to calm down the mounting concern.
Said Howell, "I was finding myself sending messages out to various executives saying 'Wait a minute. The Internet is not melting. It's certainly a serious issue, but it's not the most serious issue and there's probably some hype here.'"
With Heartbleed established as a high-priority fix, the University of Michigan security team first moved to implement intrusion detection system (IDS) filters to try to detect attacks against the vulnerability. At the same time, a separate group of computer science researchers at the university were actively scanning the open Internet for organizations exposed to Heartbleed, triggering calls from security personnel at other universities complaining that Michigan had tripped their own IDS filters, leading to a brief moratorium on the scans.
With IDS filters in place, Howell said that Michigan staff began rolling out the updated OpenSSL version with the Heartbleed patch to hundreds of internal systems, a process that he described as relatively straightforward and on par with what he heard from other universities. Some systems, such as those that handled the university's authentication processes, were actually not vulnerable; internal systems that needed patching were addressed in around four days.
Once the majority of systems had been applied, Howell said Michigan's Web administrators took the next step of revoking its previous security certificates and replacing them with new ones through InCommon -- again, a largely pain free process. In fact, only one application was broken as part of the patch rollout, according to Howell, and the university sidestepped that issue by manually disabling the TLS heartbeat extension at the root of the vulnerability.
Heartbleed patching: Who owns what?
Where the university ran into trouble, said Howell, was with outlying systems active on Michigan's network that were being managed through individual departments, mostly by graduate students and researchers. Those systems, he said, couldn't simply be patched using the same process as was being used for internal systems managed by the school's various IT staffs.
Not only did it take longer to patch those systems, according to Howell, because the IT staff tasked with handling different departments' systems quickly ran dry during an emergency -- in comparison, Michigan's central IT staff were able to "work through the night" patching central systems -- but some of the graduate students and faculty managing those systems also questioned whether Heartbleed was a serious enough issue to even warrant applying the update.
"All we did was block those systems at the border," said Howell. "It was easy enough to do, but you have to go through dozens and dozens of groups, communicating with each of them, and the effort and time that takes is not insignificant."
Apart from those outlying systems, Howell said that Michigan was also stuck waiting for patches to arrive from its third-party appliance vendors like Polycom. Howell conceded that like most everyone else in IT, vendors were caught off-guard by Heartbleed, though he still characterized their response as "very slow" and noted that it took a long time for some vendors to even tell the university when they were planning to patch.
Heartbleed password changes: The last big step
Once vendor updates arrived, Howell said Michigan was able to complete its Heartbleed patching efforts by communicating to the university community the need to change passwords.
The school handled that process by gradually sending emails out to various university groups over the course of several weeks, starting with university executives and department heads, then faculty and current students and finally alumni and incoming students. The school also ran a headline article on the University of Michigan Health System website that provided Heartbleed details, including background on the vulnerability and how to change university-related logon credentials.
Notably, Howell said that the university didn't require users to change passwords, a move that went against the grain of some security experts' advice after the flaw was publicized, in part because it wasn't widely agreed upon throughout the industry that such a large-scale effort was needed.
"There were a lot of mixed messages from different organizations, [with] many recommending password changes. You can never really be certain, so we didn't require it," said Howell. "We just sent out email to different communities here encouraging them to change their password and referenced the Heartbleed bug as the reason for it. But we didn't mandate password changes."
While Michigan's Heartbleed response effort went fairly smoothly, Howell expressed his hope that future bugs would not be disclosed in such a highly publicized fashion. Even though that level of transparency may serve to increase the general public awareness on security issues, Howell said that it could leave users unnecessarily vulnerable and force harried, uninformed patching efforts from organization IT teams.
"It's just another example of how hard it is for anybody to produce -- I don't want to say perfect code -- but code that doesn't have serious flaws in it," said Howell, who noted that huge software vendors like Microsoft and Oracle Corp. still commit faulty code similar to mistakes they made nearly a decade ago. "And it's not that they're doing a bad job; they're doing a really good job trying to produce secure code, but it's really hard to get all the bugs out."
Want more info on the Heartbleed OpenSSL vulnerability? Resident application security expert Michael Cobb details how Heartbleed changed open source software security, while Cigital CTO Gary McGraw discusses some of the lessons learned from the flaw.