Unfortunately, stealth hacking occurs because many security managers and admins aren't looking for clues or don't know what to look for. How often do your admins actually check their server logs? They're often too busy to wade through reams of log data. And, even if you've got an especially diligent admin, he may tell you, "I've seen lots of stuff, but so what? Everything is running smoothly."

But what if you came across this piece of gibberish?

viewtopic.php?t=%33%32&rush=%65%63%68%6F%20%5F%53%54%41%52%54%5F%3B%20%63%64%20%2E%2E%2F%3B%65%63%68%6F%20%44%61%72%6B%2D%55%6E%64%65%72%

Nonsense? Not At All.

It's part of the message, "Dark Underground was here." This is the wake-up call. Some hackers like to leave calling cards to boast of their work. It may seem like a simple defacement, but it could be a clue that something really bad is going on under the placid surface.

Unfortunately, you can't assume that every hacker is going to "sign" his work. Finding that covert intruder takes a combination of good logging strategy, tools and insight.

For instance, a hacker who's cracked a legitimate user account looks just like any other traffic using SSH to remotely connect to your systems. The logins look legit, but they are probably originating from external servers. And, you aren't going to notice unwanted visitors if you allow SSH logins from anywhere to your production or internal servers.

Consider restricting SSH so only trusted IP ad-dresses are allowed to make SSH connections. Also, you'll want to have your SSH sessions authenticate to the perimeter firewall so you have a record of SSH activity in ...

You should use the tools at your disposal, especially syslog. It allows you to consolidate the logs of Unix and Linux servers, workstations and most appliances to a central server. The server must be protected, typically placed behind the DMZ, and receive logs through an encrypted tunnel. Microsoft should be ashamed of itself for not natively supporting syslog on its workstations, but third-party tools will centralize Windows logs to a syslog server.

A good place to start is looking at the Unix main page for the "last" command; this will show you users who have recently logged in. Also, look at the SU log and see who has become or attempted to become a super-user. This is a bit trickier in Win-dows, since everyone typically works with administrative privileges, which makes it harder to spot privilege escalation.

Looking for interesting security events can be like searching for the proverbial needle in a haystack. Legitimate events can get overlooked when you're plowing through the endless number of IDS alarms. Unless you trace every event back to its targeted server, you won't know if the attack was a success or failure. SIM systems are useful for homing in on meaningful events and automatically sifting through logs. Even if you don't use a SIM, consider an open-source log-watching tool like Swatch. Both will cut down on time and effort.

Reviewing logs will never be glamorous or easy, but it sure beats rebuilding machines that have been hacked.