When Tatu Ylonen created the Secure Shell network protocol in 1995, he couldn't have imagined how widespread it...
would become decades later -- nor could he imagine the security risks that would come with that growth.
While Ylonen had developed SSH as a way to authenticate and secure communications over the internet, poor SSH security practices have essentially created the opposite effect for enterprises today. According to Ylonen, companies have generated so many SSH keys that they've lost track of them -- and these keys, if fallen into the wrong hands, could lead to devastating data breaches.
Ylonen, founder of SSH Communications Security, spoke with SearchSecurity recently about his creation and how the cryptographic network protocol's use has evolved and expanded over the last 20 years. He also talked about the current state of SSH security within enterprises, Akamai Technologies' recent discovery of a major OpenSSH flaw, and how his company is trying to help companies track down and better manage all of those keys before attacks take advantage of them.
Here are excerpts of the conversation with Ylonen. For the audio version of the interview, listen to this episode of the Risk & Repeat podcast.
What is your take on the current state of SSH? How has SSH use evolved over the last 10 years and how you see it being used today? And has that led to some of the problems?
Tatu Ylonen: Well, the one big change is that it comes automatically with many devices. All Linux, all Unix, all Macs, even Android has SSH that you can enable. It comes in pretty much all embedded devices. It comes in BIOS firmware, in most server hardware nowadays with IPMI [Intelligent Platform Management Interface] ports and so on. It's so everywhere. Airplane entertainment systems -- I've seen them boot from SSH. You can see the messages during the boot. It's everywhere. So, that's been a change.
Another change is that it's being used in a number of places that aren't regularly updated, like security cameras that don't have firmware update features. I think that has to change about IoT [internet of things]. I just think that society will collapse if we have devices that just start pushing data at somebody for DDoS [distributed denial-of-service] attacks. And I see a few avenues for fixing that.
One: liability. You force [device manufacturers] to do things properly and securely, or go out of business. Two: Use really good updating systems, which is part of what liability can force them to do. Just have them provide software upgrades so that the devices get them and they fix them quickly when there's a problem. Three: Brick them.
Brick the devices, you mean?
Ylonen: Yes. Use the same vulnerabilities to shut them down permanently. And that, of course, probably involves liability for vendors and so on and probably, right now, is not legal. But if you think of it from society's perspective, if it's going to shut down the networks, probably shut down telecommunications in some cases, or cause deaths because emergency services don't get on site because you can't even call 911 on your systems, we can't take that. Something has to be done.
So, what should be done first?
Ylonen: Many of the device breaches were using default passwords. That, I think, is the easiest fix for vendors to do. Don't have default passwords. Use [a] unique password for each device right on the bottom of the device. Often, security starts from pretty simple things. A unique password and operating mechanism takes you pretty far, but folks haven't done it. But SSH is being used in these devices increasingly.
The Akamai [SSHowDowN Proxy attack] report showed millions of security cameras and satellite dish control systems and so on were vulnerable because of a configuration error. It's trivial to fix if you have an upgrade mechanism. If you don't, you lose your reputation. You might have liability. I actually think that there might be reasonable theories of liability for IoT manufacturers if the device is designed using principles that are known to be very likely to cause harm to others or their owners.
But then we have SSH in data centers. It's in every data center for router management, for switch management, for virtualization management and so on. And in data centers, the biggest problem is SSH keys, which are just an authentication mechanism. You have a cryptographic key in a file, and you configure on a server. And with the key, you are able to log in without having to type a password. And that's used for all the automated processes: automated confirmation management, automated provisioning, automated audits, emergency response, copying data to disaster recovery data centers and so on.
And it's turned out those keys haven't been managed. And the access they give is basically like passwords on the operating system. They give you a shell access typically. We found in a bank 10% of SSH keys granted root access -- the highest-level administrative access, which lets you read any files, modify any files, install new kernel drivers or reprogram the firmware on the device. You can brick all the disk drives and the BIOS and the machine won't boot. And you can, theoretically, do that to tens of thousands of servers, which could affect multiple enterprises and critical infrastructure simultaneously could be disrupted.
And we went to a Wall Street bank [that] went through about 15,000 servers, which were about a quarter of the infrastructure, and 500 applications, which were some of the most critical production applications. From those, we found 3 million SSH keys. Three million, and 10% or 300,000 of those keys granted root access. Some of them granted access to disaster recovery data centers, which are supposed to take over if one data center goes down. But if somebody hacks into a data center, and this password allows the hacker access to another data center that's being used to copy data to backup systems and the hacker gets your backup system, then you're sort of screwed a bit.
Why are there so many SSH keys out there? What are they being created for?
Ylonen: In many cases, these were installed for good purposes and good intentions, like an Oracle admin just wanting to be able to log in from his personal account to the different Oracle database accounts in the enterprise and be able to do admin operations or monitoring in an automated fashion.
The trouble was those are pretty critical accounts that normally would require going through privileged access management systems and so on. And these keys bypassed all the access management systems they had in place. So, they could access the databases without recourse. And I think it's a problem, especially for a bank with such critical data: account balances, stock holdings, etc.
You mentioned how SSH has become ubiquitous. In terms of exposed SSH keys, is the lack of key management the bigger issue than the ubiquitous nature of SSH?
Ylonen: I think the biggest cause of the problem was that identity and access management professionals didn't know about SSH keys. They typically came from a Windows background, or they were kind of retrained ex-military veterans or others who didn't have the Unix admin background and didn't know that these sort of credentials exist. And it wasn't described in books. It still isn't described in most books on identity and access management. They talk about passwords and two-factor authentications, they talk about single sign-on, but they don't talk about SSH keys.
I think that's the big educational thing. Therefore, because they didn't know it, it wasn't written in the policies. And OpenSSH in default configuration allows any user who logs into an account [to] configure additional credentials for that account -- additional permanent credentials. It's the only credential that, typically, users are able to provision themselves.
Now, you can configure it to lock down those keys so that only the root of an automated system can install keys. But, by default, anybody can install new credentials. Therefore, sys admins [system administrators] install them for convenience, or they install them whenever they needed to transfer data between two information systems.
And, in most cases, it was done with good purposes, but turned out that in that bank, 90% of those 3 million keys were never used. We monitored the environment for years. And 90% of them were never used. Think of usernames and passwords that were provisioned to somebody, then that person left and went to do something else and nobody ever removed the keys.
So, they were treating SSH keys as disposable passwords, except they weren't disposing the keys?
Ylonen: Yes. One of our presales guys went to a healthcare organization recently that he consulted for 10 years ago and tried [an] old key. And it still worked. So, that's where we are. So, now, almost all large enterprises find themselves in a situation where they have hundreds of thousands, even millions of credentials that grant access to their servers. They have little visibility into where that access goes to and the sort of the connections in that access, who has those keys and who's been able to obtain those keys over the years. Yet, their systems are entirely dependent on the access and automation implemented due to these keys.
In that bank, there were about 5 million daily SSH logins, mostly automated using SSH keys. They would kick off back-end operations on other servers, like log data transfers, transaction batch transfers, configuration file transfers, data file transfers like pricing data and those sorts of things. They basically closed doors if a key was removed.
Stay tuned for part two of SearchSecurity's interview with SSH creator Tatu Ylonen.
Read more on how to address and mitigate SSH security risks
Find out how vendors are changing the state of IoT security
Learn how FIDO authentication is changing identity and access management