You've been hacked

How do you know if you've been hacked? Computer forensics, that's how.

After "You have cancer," the scariest sentence a system administrator can hear is "You've been hacked." But how do you know if you've been hacked? This is where computer forensics comes in.

Oddly enough, the first indication you've been hacked is your users. If a user comes to you saying, "Ya know, the network is running really slow today" or "How come the disk farm is out of space? I thought we added more space last month", it may be an indication you've been hacked.

What's the first step dealing with being hacked? Ideally, you reach into your desk and pull out the Policies and Procedures Manual and flip to the section on "Incident Handling". Most of us are not so lucky. If it's possible, take the system offline, remove the disk(s), make image-level copies of the disks, and store the original disks in a safe storage location. Probably the last thing you want to do is restore the system from a backup (you do have a backup) and put the system back online. This is just an open invitation to get hacked again.

Assuming you can't immediately take the potentially hacked system down, there are some forensic methods you can use to assess the state of the system. Among the first things to check are the system log files. The file /var/log/sulog records usage of the su command. Look for entries where an unauthorized user has used the command inappropriately. The following entry shows a successful (indicated by +) su from user userid to root.

SU 03/31 12:52 + pts/0 <userid>-root

Look for failed su attempts.

SU 03/31 12:52 - pts/0 <userid>-root

Look for su occurring at unusual times during the day (e.g., after hours, very early in the morning).

SU 03/31 02:12 + pts/0 <userid>-root

Next check /var/log/utmp and /var/log/utmpx. /var/log/utmp; /var/log/utmpx. These logs keep track of users currently logged into the system. Using the who command, check the users logged in at the current time:

<userid> pts/1 Oct 27 08:40 (origination hostname)

Look for user logins that are unexpected (e.g., for staff on vacation), occur at unusual times during the day, or originate from unusual locations. Logins and logouts are tracked in /var/log/wtmp and /var/log/wtmpx. Look for user logins occurring at unusual times.

<userid> pts/4 <hostname> Sat Oct 26 03:14 - 06:02 (02:47)

Look for user logins originating from unusual places (locations, addresses, and devices).

<userid> pts/12 <strange hostname> Fri Oct 25 08:59 - 13:30 (04:31)

Look for unusual reboots of the system.

reboot system boot Sun Oct 27 05:36

By default, the syslog file will contain only messages from mail (as defined in the /etc/syslog.conf file). Look for anything that looks unusual especially large gaps of time between records. This could indicate deleted data.

The file, /var/adm/messages, records system console output and syslog messages. Look for unexpected system halts.

Oct 27 12:48:41 <hostname> unix: halted by <userid>

Look for unexpected system reboots.

Oct 27 12:48:41 ahost.domain.com unix: rebooting...

Look for failed su and login commands.

Oct 26:14:00 <hostname> login: 4 LOGIN FAILURES ON 0, <userid>

Oct 27 12:37:43 <hostname> su: 'su root' failed for <userid> on /dev/pts/??

Look for unexpected successful su commands.

Oct 27 14:31:11 <hostname> su: 'su root' succeeded for <userid> on /dev/console

The file, /var/adm/pact, records the commands run by all users. Process accounting must be turned on before this file is generated. You may want to use the lastcomm command to audit commands run by a specific user during a specified time period.

compile <userid> ttyp1 0.35 secs Fri Oct 25 12:59

Don't ls and ps commands. These have probably been replaced by hacked versions that do not accurately portray files in directories or processes running. If you suspect a bogus ls, you can list the normal files in a directory with the command "echo *". Hidden files, i.e., those starting with a ".", still won't show up. You can search for hidden files with find / -name ".*" –print. In the case of a suspect ps command, you can use the crash command to compare output of ps:

crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> proc
PROC TABLE SIZE = 3898
SLOT ST  PID  PPID  PGID   SID   UID PRI   NAME        FLAGS
   0 t     0     0     0     0     0  96 sched          load sys lock
   1 s     1     0     0     0     0  58 init           load
   2 s     2     0     0     0     0  98 pageout        load sys lock 
   3 s     3     0     0     0     0  60 fsflush        load sys lock 
   4 s   212     1   212   212     0  58 sac            load jctl
   5 s   217     1   217   217     0  58 dtlogin        load jctl
   6 s   101     1   101   101     0  42 keyserv        load
   7 s    43     1    43    43     0  44 syseventd      load
   8 s    45     1    45    45     0  34 syseventconfd  load
   9 s   174     1   174   174     0  58 lpsched        load nowait
  10 s    98     1    98    98     0  58 rpcbind        load
  11 s   136     1   136   136     0  33 lockd          load
  12 s   188     1   188   188     0  59 utmpd          load
  13 s   111     1   111   111     0  58 ypbind         load
  14 s   147     1   147   147     0  48 automountd     load
  15 s   135     1   135   135     1  33 statd          load
  16 s   166     1   166   166     0  48 cron           load
  17 s   133     1   133   133     0  58 inetd          load
  18 s   193   192   192     0     0  43 smcboot        load jctl
  19 s   195     1   195   195     0   0 vold           load jctl
  20 s   164     1   164   164     0  59 syslogd        load
  21 s  1456  1454  1456  1456 20957  48 tcsh           load jctl
  22 s   192     1   192     0     0  33 smcboot        load jctl
  25 s   213     1   213   213     0  54 ttymon         load

The hacker has probably left behind setuid/setguid executables on the system. Check for setuid executables with find / -perm –4000 –print and setgid executables with find / -perm –2000 –print.

Check system files for irregularities, for instance, /etc/inet/inetd.conf. Look for changes or unfamiliar services. Look through /etc/hosts.equiv and any ,rhosts files for a "+" entry that would give unrestricted access to any system. Sometimes hackers leave text files in the /dev directory as configuration files for hacked executables. Check the /dev directory for text files with find /dev –type f –print | xargs file | grep text. The hacker may have created a backdoor for later entry into the compromised system. Examine /etc/passwd, /etc/group, and /etc/shadow for changes or additions of userids and group permissions, or password changes. As root, do crontab –l and atq to see if unusual jobs have been set up by hackers to do things such as email the password/shadow for remote password cracking.

Run a portscan against the compromised machine using nmap from a known good remote machine. Nmap is available at http://www.sunfreeware.com or http://www.insecure.org. Nmap is a very efficient program and can scan all 65000+ TCP and UDP in a matter of seconds. Do something like nmap –p 1-65535 rhett.gwtw.org. Compare that against the output of netstat –a. If ports show up in the nmap trace that do not show up in the netstat listing, it's likely the local version of netstat has been compromised to hide rogue services left running by the hacker.

Finally, Sun has produced a unique online tool enabling cryptographic comparison of system binaries, the The Solaris Fingerprint Database. This is available at http://sunsolve.sun.com/pub-cgi/fileFingerprints.pl. With this you can verify that suspect executables match the Sun distribution.

In conclusion, being hacked is a scary experience. With some forensic analysis, you can be on your way to determining what happened and what was done. The steps to take will vary with your site but at the least:

· Check the log files for unusual activity or gaps.
· Check for unusual files or processes.
· Check for backdoors.
· Check for unusual network services.
· Check the fingerprint of suspect executables against known good fingerprints.


This was first published in December 2002

Dig deeper on Hacker Tools and Techniques: Underground Sites and Hacking Groups

Pro+

Features

Enjoy the benefits of Pro+ membership, learn more and join.

0 comments

Oldest 

Forgot Password?

No problem! Submit your e-mail address below. We'll send you an email containing your password.

Your password has been sent to:

-ADS BY GOOGLE

SearchCloudSecurity

SearchNetworking

SearchCIO

SearchConsumerization

SearchEnterpriseDesktop

SearchCloudComputing

ComputerWeekly

Close