The Secure Shell network protocol is so widely used today that you can find in just about any connected device,...
including internet of things devices, industrial control systems and massive internet-connected machinery. And that, according Tatu Ylonen, brings major security risks that could move beyond cyberspace and into the physical world.
Ylonen, creator of the Secure Shell (SSH) protocol, spoke with SearchSecurity about the dangers of IoT SSH implementations and why companies need to do a better job managing their SSH keys. If attackers obtain an SSH key file, they could use those keys to gain unfettered access to industrial control systems (ICS) and even utility company operations, which could have major consequences, he said.
Ylonen, founder of SSH Communications Security, talked about how IoT SSH implementations work and the risks they carry, as well as real-world examples of enterprises that have had their keys exposed. He also discussed how attackers are using SSH tunnels to establish persistent access into enterprise and government environments, and explained why it's hard to spot those tunnels.
Here are excerpts of the conversation with Ylonen. For the audio version of the interview, listen to this episode of the Risk & Repeat podcast. Read more on the topic with parts one and two of this interview series, which cover the evolution of SSH use in the enterprise and improper SSH key management.
You mentioned how enterprise rely a lot on SSH in their data centers. What about industrial control systems and IoT devices? Is that an area where you're seeing a lot of IoT SSH keys being used, but not properly managed?
Ylonen: I'm seeing it used by field services organizations. I've had a couple of customer cases, and one was a company that makes harbor cranes -- these big, multimillion-dollar machines that could do massive damage if they go wild. And the field services organization logs into what is basically a PC that's in the crane, and it does preventative maintenance, monitoring, upgrades and so on. And in that case they installed crypto for monitoring those connections, and they get visibility into what the field service organization does with those machines.
It's in part a liability limitation. So in another case kind of similar we had a customer that was making storage systems and, again, the field services organization logs into customer systems from the outside using SSH to manage and upgrade and monitor those storage devices. And there again they used [SSH] crypto to get visibility into that process.
SSH is in the hardware of many of these systems, so maybe the manufacturers who made the original DVRs or ICSs knew about them then, but do most people who work there know? Do they even know that these things are possible?
Tatu Ylonen: My guess would be that they just didn't think of it. And they were under time pressures as people always are. There are different problems in the smaller IoT devices, but then there's the really big, really heavy industrial IoT, like large engines of ships or engines in factories or paper mills and really big machinery. They are increasingly serviced by the manufacturer and [manufacturers are] increasingly wanting to optimize the process. They're constantly developing analytics and control technologies for those, so there's improvement in maintenance. And there's a lot of pressure to minimize service visits often to remote locations into those factories. So it's all remotely managed and monitored. And it's both in the factories' and the vendors' interest to keep optimizing those and monitoring those machines. Any downtime is very expensive because these are heavy capital investments.
But the damage also could be significant. And it's going to apply to utilities increasingly. We have some customers in the energy and utility space. They are increasing the network decentralized. The Ukrainian attacks where they brought down the Ukrainian power grid also involved SSH. There was an SSH implementation embedded into the malware they used for that attack.
Was that the BlackEnergy malware?
Ylonen: Yes. [SSH] was basically use of a backdoor in that case.
When an organization stores these keys, presumably there's a file or directory where they keep those things. If somebody breaks into a system, could they get access to the file system? I'm assuming that there's a way to write a script that'll just automate and send back a list of all the systems that can be accessed by those keys.
Ylonen: Our software automates a lot of the maintenance process, like renewals and discovery [of SSH keys]. It just scans the whole environment, correlates them between different systems, reports and policies. There's information about the reporting function that we've improved a lot on the compliance reporting so that we can automatically detect things like violations and segregation of duties, which is pretty important. For example, access from developer systems into production systems -- development systems don't have the same protections, so we don't want somebody to be able to log from development systems into production systems without the password. There was loads of this in most banks we work with, even in the most sacred environments, and it's probably even more chaotic in the average enterprise.
But we'd never copy private keys into our system. We only take public keys. So even if an attacker gets them, they are not very useful. There are deployments scenarios where our software may have a private key that can be used to log into another system to manage that. And we can store those keys in an HSM [hardware security module].
But let's assume your software isn't there. An attacker breaks into the system, gets the root privilege, and looks at the SSH configurations. Are they going to be able to identify which of those credentials relate to which systems? In other words, how easy is it once you get into a system to pivot to the other 199 systems in that environment?
Ylonen: It's fairly easy. They would look at syslog data or they would just sit on the system for a few weeks and look at the process table every few seconds. It's pretty easy, especially if you have time and you're, let's say, preparing for a cyberwarfare scenario where you want to gain access to as many places as possible and remain hidden. All about intelligence and preparation and then once you press the button things happen very fast. And so in those cases you can sit there for months, and you just watch and you just collect information. From the process table you can see every SSH connection that exists going out. And you can see the power meters, and you can pretty much see what keys they use. And that can go on undetected for months. So [the attacker] has that information and may even have tested those connections but not used them too much because that would be detectable by IDS [intrusion detection systems] and so on. But they just quietly watch and once it's time to do something, then it can be done very quickly. If the intelligence is done previously, then the attack can, with very high success rate, go throughout the systems in a minute and even seconds. One attempt takes about 10 milliseconds, and you can do several in parallel. And it's distributed so it's one second and you might be throughout the entire network. You've gone as far as you can. And it's faster than any human mitigation reaction. And you don't really want to have automated responses for these systems because there are just too many false positives. So the security manager turns his head and looks at the screen in the SOC [security operations center], and it's over.
Are attackers doing this sort of thing today?
Ylonen: Talk to penetration testers. The first thing they do is they grab all the SSH keys. It's built into Metasploit and other penetration testing tools. In the Sony breach, they took SSH key files. We've seen SSH key files being sold on black markets for bitcoins like the [bitcoin exchange platform company] ShapeShift breach, where an insider sold SSH keys for the organization for bitcoins to another hacker, who then went in and stole a much higher amount of bitcoins from the organization. You can also conceivably buy SSH tunnels. Basically, you just have somebody run SSH to a server in the cloud with certain options and suddenly anybody connecting to that cloud server actually gets tunneled over the encrypted channel, through the firewall to the internet and then connects to a server. And that also can be left behind and stay alive for even years, automatically restarting with a 10-line shell script. And since it's encrypted, typically these tunnels are pretty undetectable.
I'm sure the National Security Agency loves that.
Ylonen: You may have heard of the NIST guidelines on SSH keys published in late 2015: NISTIR 7966. And that publication was delayed by more than a year, but it seems to me like there was no reason for the delay. And I was told by people from the government that there are parties in the U.S. government that didn't want to see it published yet because it's too valuable for them. But it was finally published [in October of 2015]. The NIST cybersecurity group knew almost three years ago already that this is something it needed to understand. I had calls with people from the White House and from NSA, and from Cisco, Juniper and some of the other key vendors the federal government uses, and the view was that SSH is massively used within government systems.
Read more on how to approach an SSH security risk assessment
Find out how to monitor outbound traffic and what to look for
Discover how web application firewalls prevent application attacks