Detecting a Linux server hack

Learn how to detect if your Linux server have been hacked or compromised.

Q: How can I determine if my Linux server has been hacked? How can I be sure that I haven't been hacked? -G.C.

A: Being hacked is a lot like being haunted--odd things are afoot that you may or may not notice. That said, simple observation is the easiest way to detect when it's happened. For example, extra users running around your system--from the obvious second superuser root account to the "sneaky john" account that you never created--are easy to spot if you're keeping an eye on things. You might also observe file changes that you never made, or programs running that you never started--such as a sniffer, an IRC program or a file-sharing program.

Beyond simple observation, my first tool of choice for detecting server intrusions is the freeware Linux version of Tripwire, which checks files to see if they've been altered, either in their contents or metadata (ownership, permissions, etc.). Tripwire's an excellent tool for detecting break-ins, which often involve changes to "critical" system files. You have to run Tripwire at least once to generate a baseline of your system's critical files. This baseline includes stored metadata for each file, along with a "fingerprint" constructed from its contents.

Tripwire isn't infallible. A hacker can alter your kernel to mislead any file integrity tool. While Tripwire can detect an outright substitution of your SSH program, the Knark rootkit can use "execution redirection" to force your system to run /tmp/grab-my-password whenever a user tries to run SSH. The "grab-my-password" command might be a modified version of SSH that recorded your password before making the SSH connection. Because Knark leaves SSH in place, simply redirecting execution to grab-my-password, Tripwire wouldn't detect the substitution.

Tripwire's not failing here; the attacker just altered the kernel and changed the fundamental nature of the system. To avoid this problem, you can boot the system from a clean floppy disk or CD-ROM before running the Tripwire checks, ensuring a clean kernel.

OK, but what if a clean reboot isn't possible?

In that case, bring in chkrootkit, a tool that searches for particular oddities introduced into the system by rootkits, often by mistakes made in the rootkits' designs. Some mistakes involve flawed program replacements that don't respond properly to certain less common command-line arguments. For this reason, chkrootkit can often detect rootkits on systems with a compromised kernel. Additionally, it's easy to use and requires no pregenerated baselines.

But chkrootkit also has its limitations. It will detect only known rootkits. For instance, chkrootkit will never notice a custom backdoor. The good news is that Tripwire usually will.

How else can you detect that you've been hacked? Well, many hackers leave a specialized backdoor to allow easy re-entry, even after you patch the vulnerability that originally allowed them in. Backdoors often force the computer to listen on an extra network port for external connections. You can ask a system what ports it's listening on by using the netstat command.

This isn't a foolproof method for detecting backdoors; the computer could be set up, via a rootkit, to lie to you about the listening network port. Nmap and similar programs can remotely catalog the system's listening ports. Of course, you have to already have a good idea what ports should be listening on the system. Otherwise, you'll have to investigate every port to determine if it should be open.

Even with all this, there's no way to be sure that you haven't been hacked. A hacker could compromise a system in ways that involve no changes to any critical files, which would elude Tripwire. He may not install rootkits, rendering chkrootkit useless. And he might not install network listeners, instead exploiting a unique vulnerability for access every time he revisits.

If you need absolute certainty, you're usually stuck reinstalling the OS.

Really, the secret is to avoid compromise. Harden systems before deployment, keep up with patches, and design a strong host and network architecture. Then start building your own digital and human baselines to make intrusion detection easier. It's a bit of upfront work, but it's less hassle than recovering from a bad compromise later.

About the author:
Jay Beale is lead developer of the Bastille Linux project, which creates a lockdown script for five Linux distributions and HP-UX.

Dig Deeper on Network intrusion detection and prevention (IDS-IPS)