After "You have cancer," the scariest sentence a system administrator can hear is "You've been hacked." But how do you know if you've been hacked? This is where computer forensics comes in.
Oddly enough, the first indication you've been hacked is your users. If a user comes to you saying, "Ya know, the network is running really slow today" or "How come the disk farm is out of space? I thought we added more space last month", it may be an indication you've been hacked.
What's the first step dealing with being hacked? Ideally, you reach into your desk and pull out the Policies and Procedures Manual and flip to the section on "Incident Handling". Most of us are not so lucky. If it's possible, take the system offline, remove the disk(s), make image-level copies of the disks, and store the original disks in a safe storage location. Probably the last thing you want to do is restore the system from a backup (you do have a backup) and put the system back online. This is just an open invitation to get hacked again.
Assuming you can't immediately take the potentially hacked system down, there are some forensic methods you can use to assess the state of the system. Among the first things to check are the system log files. The file /var/log/sulog records usage of the su command. Look for entries where an unauthorized user has used the command inappropriately. The following entry shows a successful (indicated by +) su from user userid to root.
SU 03/31 12:52 + pts/0 <userid>-root
Look for failed su attempts.
SU 03/31 12:52 - pts/0 <userid>-root
Look for su occurring at unusual times during the day (e.g., after hours, very early in the morning).
SU 03/31 02:12 + pts/0 <userid>-root
Next check /var/log/utmp and /var/log/utmpx. /var/log/utmp; /var/log/utmpx. These logs keep track of users currently logged into the system. Using the who command, check the users logged in at the current time:
<userid> pts/1 Oct 27 08:40 (origination hostname)
Look for user logins that are unexpected (e.g., for staff on vacation), occur at unusual times during the day, or originate from unusual locations.
Logins and logouts are tracked in /var/log/wtmp and /var/log/wtmpx. Look for user logins occurring at unusual times.
<userid> pts/4 <hostname> Sat Oct 26 03:14 - 06:02 (02:47)
Look for user logins originating from unusual places (locations, addresses, and devices).
<userid> pts/12 <strange hostname> Fri Oct 25 08:59 - 13:30 (04:31)
Look for unusual reboots of the system.
reboot system boot Sun Oct 27 05:36
By default, the syslog file will contain only messages from mail (as defined in the /etc/syslog.conf file). Look for anything that looks unusual especially large gaps of time between records. This could indicate deleted data.
The file, /var/adm/messages, records system console output and syslog messages. Look for unexpected system halts.
Oct 27 12:48:41 <hostname> unix: halted by <userid>
Look for unexpected system reboots.
Oct 27 12:48:41 ahost.domain.com unix: rebooting...
Look for failed su and login commands.
Oct 26:14:00 <hostname> login: 4 LOGIN FAILURES ON 0, <userid>
Oct 27 12:37:43 <hostname> su: 'su root' failed for <userid> on /dev/pts/??
Look for unexpected successful su commands.
Oct 27 14:31:11 <hostname> su: 'su root' succeeded for <userid> on /dev/console
The file, /var/adm/pact, records the commands run by all users. Process accounting must be turned on before this file is generated. You may want to use the lastcomm command to audit commands run by a specific user during a specified time period.
compile <userid> ttyp1 0.35 secs Fri Oct 25 12:59
Don't ls and ps commands. These have probably been replaced by hacked versions that do not accurately portray files in directories or processes running. If you suspect a bogus ls, you can list the normal files in a directory with the command "echo *". Hidden files, i.e., those starting with a ".", still won't show up. You can search for hidden files with
find / -name ".*" –print. In the case of a suspect ps command, you can use the crash command to compare output of ps:
crash
dumpfile = /dev/mem, namelist = /dev/ksyms, outfile = stdout
> proc
PROC TABLE SIZE = 3898
SLOT ST PID PPID PGID SID UID PRI NAME FLAGS
0 t 0 0 0 0 0 96 sched load sys lock
1 s 1 0 0 0 0 58 init load
2 s 2 0 0 0 0 98 pageout load sys lock
3 s 3 0 0 0 0 60 fsflush load sys lock
4 s 212 1 212 212 0 58 sac load jctl
5 s 217 1 217 217 0 58 dtlogin load jctl
6 s 101 1 101 101 0 42 keyserv load
7 s 43 1 43 43 0 44 syseventd load
8 s 45 1 45 45 0 34 syseventconfd load
9 s 174 1 174 174 0 58 lpsched load nowait
10 s 98 1 98 98 0 58 rpcbind load
11 s 136 1 136 136 0 33 lockd load
12 s 188 1 188 188 0 59 utmpd load
13 s 111 1 111 111 0 58 ypbind load
14 s 147 1 147 147 0 48 automountd load
15 s 135 1 135 135 1 33 statd load
16 s 166 1 166 166 0 48 cron load
17 s 133 1 133 133 0 58 inetd load
18 s 193 192 192 0 0 43 smcboot load jctl
19 s 195 1 195 195 0 0 vold load jctl
20 s 164 1 164 164 0 59 syslogd load
21 s 1456 1454 1456 1456 20957 48 tcsh load jctl
22 s 192 1 192 0 0 33 smcboot load jctl
25 s 213 1 213 213 0 54 ttymon load
The hacker has probably left behind setuid/setguid executables on the system. Check for setuid executables with find / -perm –4000 –print and setgid executables with find / -perm –2000 –print.
Check system files for irregularities, for instance, /etc/inet/inetd.conf. Look for changes or unfamiliar services. Look through /etc/hosts.equiv and any ,rhosts files for a "+" entry that would give unrestricted access to any system. Sometimes hackers leave text files in the /dev directory as configuration files for hacked executables. Check the /dev directory for text files with find /dev –type f –print | xargs file | grep text. The hacker may have created a backdoor for later entry into the compromised system. Examine /etc/passwd, /etc/group, and /etc/shadow for changes or additions of userids and group permissions, or password changes. As root, do crontab –l and atq to see if unusual jobs have been set up by hackers to do things such as email the password/shadow for remote password cracking.
Run a portscan against the compromised machine using nmap from a known good remote machine. Nmap is available at http://www.sunfreeware.com or http://www.insecure.org. Nmap is a very efficient program and can scan all 65000+ TCP and UDP in a matter of seconds. Do something like nmap –p 1-65535 rhett.gwtw.org. Compare that against the output of netstat –a. If ports show up in the nmap trace that do not show up in the netstat listing, it's likely the local version of netstat has been compromised to hide rogue services left running by the hacker.
Finally, Sun has produced a unique online tool enabling cryptographic comparison of system binaries, the The Solaris Fingerprint Database. This is available at http://sunsolve.sun.com/pub-cgi/fileFingerprints.pl. With this you can verify that suspect executables match the Sun distribution.
In conclusion, being hacked is a scary experience. With some forensic analysis, you can be on your way to determining what happened and what was done. The steps to take will vary with your site but at the least:
· Check the log files for unusual activity or gaps.
· Check for unusual files or processes.
· Check for backdoors.
· Check for unusual network services.
· Check the fingerprint of suspect executables against known good fingerprints.
