The following is an excerpt from the book Malware Forensics Field Guide for Linux Systems: Digital Forensics Field Guides written by Cameron H. Malin, Eoghan Casey and James M. Aquilina and published by Syngress. This section from chapter three explains how forensic examination of Linux systems is like an autopsy of a computer impacted by malware, and outlines a repeatable approach to conducting forensic examinations in malware incidents.
Employing a methodical approach to examining areas of the compromised system that are most likely to contain traces of malware installation and use increases the chances that all traces of a compromise will be uncovered, especially when performed with feedback from the static and dynamic analysis covered in Chapters 5 and 6.
Search for Known Malware
Use characteristics from known malware to scour the file system for the same or similar items on the compromised computer.
Many intruders will use easily recognizable programs such as known rootkits, keystroke monitoring programs, sniffers, and anti-forensic tools (e.g., touch2, shsniff,
sshgrab
). There are several approaches to locating known malware on a forensic duplicate of a compromised computer.
rpm -Va
on Linux is designed to verify all packages that were installed using RedHat Package Manager. For instance, the results of this verification process in the T0rnkit scenario are shown in Figure 3.3 to show binaries that have different filesize (S), mode (M), and MD5 (5) than expected. Some of these binaries also have discrepancies in the user (U), group (G), and modified time (T). With rpm
it is also possible to specify a known good database using the --dbpath
option, when there are concerns that the database on the subject system is not trustworthy.# rpm –Va -–root=/mntpath/evidence | grep SM5
SM5..UG. /sbin/syslogd
SM5..UG. /usr/bin/find
SM5....T c /etc/conf.linuxconf
SM5..UG. /usr/sbin/lsof
SM5..UG. /bin/netstat
SM5..UG. /sbin/ifconfig
SM5..UGT /usr/bin/ssh
SM5..UG. /usr/bin/slocate
SM5..UG. /bin/ls
SM5..UG. /usr/bin/dir
SM5..UG. /usr/bin/md5sum
SM5..UG. /bin/ps
SM5..UG. /usr/bin/top
SM5..UG. /usr/bin/pstree
SM5....T c /etc/ssh/sshd_config
FIGURE 3.3 - T0rnkit rootkit files found using RPM verify
# rkhunter --check -r /media/_root -l
/evidence/rkhunter.log
[ Rootkit Hunter version 1.3.8 ]
Checking system commands...
Performing 'strings' command checks
Checking 'strings' command [ OK
]
Performing file properties checks
Checking for prerequisites [ Warning ]
/media/_root/sbin/chkconfig [ Warning ]
<excerpted for brevity>
Checking for rootkits...
Performing check of known rootkit files and directories
55808 Trojan - Variant A [ Not found
]
ADM Worm
[ Not found ]
AjaKit Rootkit
[ Not found ]
Adore Rootkit
[ Warning ]
Performing additional rootkit checks
Suckit Rookit additional checks [ OK ]
Checking for possible rootkit files [ Warning ]
Checking for possible rootkit strings [ Warning ]
=====================
Rootkit checks...
Rootkits checked : 227
Possible rootkits: 3
Rootkit names : Adore, Tuxtendo, Rootkit component
One or more warnings have been found while checking the system.
Please check the log file (/evidence/rkhunter.log)
FIGURE 3.4 - Scanning a target drive image with rkhunter
# clamscan –d /examination/clamdb -r -i –l
clamscan.log /mnt/evidence
----------- SCAN SUMMARY -----------
Known viruses: 1256684
Engine version: 0.97.3
Scanned directories: 20
Scanned files: 46
Infected files: 1
Data scanned: 0.29 MB
Data read: 3340.26 MB (ratio 0.00:1)
Time: 6.046 sec (0 m 6 s)
FIGURE 3.5 - Clam AntiVirus software scanning a mounted forensic duplicate
frag_find
can be used to search for parts of the reference dataset on the compromised system. In addition, a piecewise comparison tool such as ssdeep
may reveal malware files that are largely similar with slight variations. Using the matching mode, with a list of fuzzy hashes of known malware, may find specimens that are not detected with an exact hash match or by current anti-virus definitions (e.g., when embedded IP addresses change).Given the prevalence of security monitoring software, it is advisable to review any logs that were created by AntiVirus software or other programs that were running on the compromised system for indications of malware. Many AntiVirus programs have logging and quarantine features that can provide information about detected malware. When a system is running Tripwire or other system integrity checking tools that monitor the system for alterations, daily reports might exist showing which files were added, changed, and deleted during a malware incident.
proc_hackinit
and proc_istrojaned
, fp_hack
, hack_list
and proc_childofhidden
, which demonstrates that “trojan,” “hack,” and “hidden” may be useful keywords when investigating some malware incidents.from_gid•getgrgid•bad_user_access_length•openproc•opendir•closeproc•closedir•freeproc•status2proc•sscanf•stat2proc•strrchr•statm2proc•nulls2sep•file2str•file2strvec•readproc•readdir•strcat•proc_istrojaned•ps_readproc•look_up_our_self•getpid•LookupPID•readproctree•readproctab•freeproctab•list_signals•stdout•_IO_putc•get_signal•get_signal2•status•uptime•_exit•lseek•Hertz•four_cpu_numbers•loadavg•meminfo•read_total_main•procps_version•display_version•sprint_uptime•time•localtime•setutent•getutent•endutent•av•print_uptime•pname•hname•proc_addpid•pidsinuse•pids•pid•proc_hackinit•xor_buf•h_tmp•fp_hack•tmp_str•fgets•hack_list•strp•strtok•proc_childofhidden•libc.so.6•___brk_addr•__curbrk•__environ•atexit•_etext•_edata•__bss_start•_end•libproc.so.2.0.6•GLIBC_2.1•GLIBC_2.0
FIGURE 3.6 - Extract from a trojanized shared library (/lib/libproc.so.2.0.6) with unusual function names
Investigative Considerations
Authors: Cameron H. Malin, Eoghan Casey, James M. Aquilina
Learn more about Malware Forensics Field Guide for Linux Systems from publisher Syngress.
At checkout, use discount code PBTY14 for 25% off
/etc/ khubd.p2
directory contained files related to the Phalanx2 rootkit shown in Figure 3.7. However, every part of the rootkit and hidden directory is subject to change in later versions of Phalanx2, including the location and names of files.-rw-r--r-- 1 root
root 1356 Jul 24 19:58 .p2rc
-rwxr-xr-x 1 root root 561032
Jul 24 19:58 .phalanx2*
-rwxr-xr-x 1 root root 7637
Jul 28 15:04 .sniff*
-rw-r--r-- 1 root 53746 1063
Jul 24 20:56 sshgrab.py
FIGURE 3.7 - Phalanx2 rootkit and TTY sniffer components located in a hidden directory
Review the programs that are installed on the compromised system for potentially malicious applications.
Surveying the names and installation dates of programs and executable files that were installed on the compromised computer may reveal ones that are suspicious, as well as legitimate programs that can be used to gain remote access or to facilitate data theft.
/var/lib/dpkg/status
file contains details about installed packages and the /var/log/dpkg.log
file records information when a package is installed. For instance, entries in the dpkg.log file on an Ubuntu system revealing that nmap
was installed are shown in Figure 3.8. On RedHat and related Linux distributions the rpm -qa --root=/mntpath/var/lib/rpm
command will list the contents of an RPM database on a subject systems.# tail -15 /mntpath/var/log/dpkg.log
2012-06-12 14:48:20 startup archives unpack
2012-06-12 14:48:22 install nmap <none> 5.21-1.1
2012-06-12 14:48:22 status half-installed nmap 5.21-1.1
2012-06-12 14:48:23 status triggers-pending man-db 2.6.0.2-2
2012-06-12 14:48:23 status half-installed nmap 5.21-1.1
2012-06-12 14:48:23 status unpacked nmap 5.21-1.1
2012-06-12 14:48:23 status unpacked nmap 5.21-1.1
2012-06-12 14:48:23 trigproc man-db 2.6.0.2-2 2.6.0.2-2
2012-06-12 14:48:23 status half-configured man-db 2.6.0.2-2
2012-06-12 14:48:27 status installed man-db 2.6.0.2-2
2012-06-12 14:48:28 startup packages configure
2012-06-12 14:48:28 configure nmap 5.21-1.1 <none>
2012-06-12 14:48:28 status unpacked nmap 5.21-1.1
2012-06-12 14:48:28 status half-configured nmap 5.21-1.1
2012-06-12 14:48:28 status installed nmap 5.21-1.1
FIGURE 3.8 - Log entries (/var/log/dpkg.log) showing installation of potentially malicious program (nmap) on a Debian-based Linux system (Ubuntu)
/usr/local
and /opt
may reveal other applications that have been compiled and installed from source code. On RedHat and related Linux distributions the command find /mntpath/sbin –exec rpm -qf {} \; |grep "is not"
command will list all executables in the /sbin
directory on a mounted forensic duplicate that are not associated with a package.Not all installed programs will be listed by the above commands because intruders might put executables in unexpected locations. Therefore, it may be necessary to look for recently installed programs that coincide with the timing of the malware incident, or use clues from other parts of the investigation to focus attention on potentially suspicious applications. In addition, look for executable files in user home directories and other locations that are commonly accessed by users but that do not normally contain executables.
Investigative Considerations
Look for references to malware in the various startup locations on compromised systems to determine how malware managed to remain running on a Linux system after reboots.
To remain running after reboots, malware is usually relaunched using some persistence mechanism available in the various startup methods on a Linux system, including services, drivers, scheduled tasks, and other startup locations.
/var/spool/cron/crontabs
and /var/spool/cron/atjobs
configuration files./etc/inittab
calls other scripts such as rc.sysinit
and various startup scripts under the /etc/rc.d/
directory, or /etc/rc.boot/
in some older versions. On other versions of Linux, such as Debian, startup scripts are stored in the /etc/init.d/
directory. In addition, some common services are enabled in /etc/inetd.conf
or /etc/xinetd/
depending on the version of Linux. Digital investigators should inspect each of these startup scripts for anomalous entries. For example, in one intrusion, the backdoor was restarted whenever the compromised system rebooted by placing the entries in Figure 3.10 at the end of the /etc/rc.d/rc.sysinit
system startup file. The Phalanx2 rootkit is launched from a separate startup script under the /etc/rc.d/ directory with the same randomly generated name as the hidden directory where the rootkit components are stored. Be warned that Phalanx2 also hides the startup script from users on the system, making forensic examination of the file system an important part of such malware investigations.# Xntps (NTPv3 daemon) startup..
/usr/sbin/xntps –q
# Xntps (NTPv3 deamon) check..
/usr/sbin/xntpsc 1>/dev/null 2>/dev/null
FIGURE 3.10 - Malicious entries in /etc/rc.d/rc.sysinit file to restart backdoor on reboot
/lib/modules/’uname -r'
and /etc/modprobe.d
directories, and the /etc/modprobe
or /etc/modprobe.con
file. These areas should be inspected for items that are related to malware./etc/profile.d
directory and the /etc/profile
and /etc/bash.bashrc
files are executed when any user account logs in and may be of interest in malware incident. In addition, each user account has individual configuration files (∼/.bashrc
, ∼/.bash_profile
and ∼/.config/autostart
) that can contain entries to execute malware when a specific user account logs into the system.Investigative Considerations
Look in all available log files on the compromised system for traces of malicious execution and associated activities such as creation of a new service.
Linux systems maintain a variety of logs that record system events and user account activities. The main log on a Linux system is generally called messages
or syslog
, and the security log records security-specific events. Some Linux systems also have audit subsystems (e.g., SELinux) configured to record specific events such as changes to configuration files. The degree of detail in these logs varies, depending on how logging is configured on a given machine.
grep
and Splunk with the ability to filter on specific types of events.Certain attacks create distinctive patterns in logs that may reveal the vector of attack. For instance, buffer overflow attacks may cause many log entries to be generated with lengthy input strings as shown in Figure 3.11 from the messages log.
Apr 8 07:47:26 localhost SERVER[5151]: Dispatch_input: bad
request line
'BBàóÿ¿áóÿ¿âóÿ¿ãóÿ¿XXXXXXXXXXXXXXXXXX000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000004800000001073835088security0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000061Û1É1À°FÍ€‰å1Ò²f‰Ð1ɉËC‰]øC‰]ôK‰MüMôÍ€1ȉEôCf‰]ìfÇEî^O'‰Mð▯Eì‰EØÆEü^P‰Ð▯MôÍ€‰ÐCCÍ€‰ÐCÍ€‰Ã1ɲ?‰ÐÍ€‰ÐAÍ€ë^X^‰u^H1À^F^G‰E^L°^K‰ó▯M^H▯U^LÍ€èãÿÿÿ/bin/sh'
FIGURE 3.11 - Log entry showing buffer overflow attack against a server to launch a command shell
This log entry shows the successful buffer overflow had “/bin/sh” at the end, causing the system to launch a command shell that the intruder used to gain unauthorized access to the system with root level privileges.
∼/.mozilla/firefox
directory for each user account..bash_history
, .history
, .sh_history
). Figure 3.12 shows a command history from a Linux system that had its entire hard drive copied over the network using netcat. Although entries in a command history file are not time stamped (unless available in memory dumps as discussed in Chapter 2), it may be possible to correlate some entries with the last accessed dates of the associated executables, in an effort to determine when the events recorded in the command history log occurred. Some Linux systems maintain process accounting (pacct
) logs, which can be viewed using the lastcomm
command. These logs record every command that was executed on the system along with the time and user account.tcp_wrappers
) function at the packet level, catching each packet before it is processed by higher level applications and, therefore, may be configured to create very detailed logs of malicious activities on a compromised system./var/log/clamav/
for ClamAV), and any quarantined items may still be stored by the AntiVirus software in a holding area.abrt
service can capture information about programs that crashed and produced debug information. When abrtd
traps a crashing program, it creates a file named coredump
(under /var/spool/abrt
by default) containing memory contents from the crash, which may provide useful information such as attacker IP addresses.Investigative Considerations
Download the PDF of chapter three to learn more.
dmesg
, kern.log
, klog
) can show that a particular service crashed repeatedly, potentially indicating that an unstable trojanized version was installed.In some enterprise environments, syslog servers are relied on to capture logging and so local security event logging is sparse on individual Linux computers. Given the volume of logs on a syslog server, there may be a retention period of just a few days and digital investigators must preserve those logs quickly or risk losing this information.
Verify that all accounts used to access the system are legitimate accounts and determine when these accounts were used to log onto the compromised system.
Look for the unauthorized creation of new accounts on the compromised system, accounts with no passwords, or existing accounts added to Administrator groups.
/etc/passwd
, /etc/
shadow and security logs for unusual names or accounts created and/or used in close proximity to known unauthorized events./etc/sudoers
files for unexpected accounts being granted administrative access and check /etc/
groups for unusual groups and for user accounts that are not supposed to be in local or domain-level administrator groups. In addition, consult with system administrators to determine whether a centralized authorization mechanism is used (e.g., NIS, Kerberos).Investigative Considerations
Combine a review of user accounts with a review of Linux security logs on the system to determine logon times, dates of account creation, and other activities related to user account activity on the compromised system. This can reveal unauthorized access, including logons via SSH or other remote access methods
Explore the file system for traces left by malware.
File system data structures can provide substantial amounts of information related to a malware incident, including the timing of events and the actual content of malware. Various software applications for performing forensic examination are available but some have significant limitations when applied to Linux file systems. Therefore, it is necessary to become familiar with tools that are specifically designed for Linux forensic examination, and to double check important findings using multiple tools. In addition, malware is increasingly being designed to thwart file system analysis. Some malware alter date-time stamps on malicious files to make it more difficult to find them with time line analysis. Other malicious code is designed to only store certain information in memory to minimize the amount of data stored in the file system. To deal with such anti-forensic techniques, it is necessary to pay careful attention to time line analysis of file system date-time stamps and to files stored in common location where malware might be found.
mactime
histogram feature in the Sleuth Kit to find spikes in activity as shown in Figure 3.13. The output of this command shows the most file system activity on April 7, 2004, when the operating system was installed, and reveals a spike in activity on April 8, 2004, around 07:00 and 08:00, which corresponds to the installation of a rootkit.# mactime -b /tornkit/body -i hour index.hourly
04/01/2004-
04/30/2004
Hourly Summary for Timeline of /tornkit/body
Wed Apr 07 2004 09:00:00: 43511
Wed Apr 07 2004 13:00:00: 95
Wed Apr 07 2004 10:00:00: 4507
Wed Apr 07 2004 14:00:00: 4036
Thu Apr 08 2004 07:00:00: 6023
Thu Apr 08 2004 08:00:00: 312
FIGURE 3.13 - Histogram of file system date-time stamps created using mactime
/usr/sbin
and /sbin
directories for files with date-time stamps around the time of the incident, scripts that are not normally located in these directories (e.g., .sh or .php scripts), or executables not associated with any known application (hash analysis can assist in this type of review to exclude known files)./dev
directory are special files that refer to a block or character device (containing a “b” or “c” in the file permissions), digital investigators may find malware by looking for normal (non-special) files and directories./bin/sh
on a system to allow them root level access at a later time. Digital investigators can use the following commands to find setuid root files on the entire file system: find /mnt/evidence -user root -perm -04000
–print
/dev
or /tmp
), an inspection of other files in that directory may reveal additional malware, sniffer logs, configuration files, and stolen files./bin
or /sbin
directories, or if an application was replaced with a trojanized version, the inode number may appear as an outlier because the new inode number would not be similar to inode numbers of the other, original files./dev
directory in the left window pane with entries listed in the order that they exist within the directory file rather than ordered alphabetically (the tyyec
entry was added last and contains adore rootkit files). In this situation, the fact that the directory is last can be helpful in determining that it was created recently, even if date-time stamps have been altered using anti-forensic methods.Investigative Considerations
jls
and jcat
utilities in TSK.Scour files associated with applications for traces of usage related to malware.
Linux systems do not have a central repository of information like the Windows Registry, but individual applications maintain files that can contain traces of activities related to malicious activities. Some common examples of applications traces are summarized below.
∼/.ssh/authorized_keys
and ∼/.ssh/known_keys
). These entries can reveal the hostname or IP address of the remote hosts as shown in Figure 3.16.∼/.recently-used.xbel
file that contains information about files that were recently accessed using applications running in the Gnome desktop.∼/.viminfo
file that contains details about the use of VIM, including search string history and paths to files that were opened using vim.∼/.mysql_history
file that contains queries executed using MySQL.∼/.lesshst
file that contains details about the use of less, including search string history and shell commands executed via less.Investigative Considerations
Search for distinctive keywords each time such an item is uncovered during forensic analysis.
Searching for keywords is effective when you know what you are looking for but do not know where to find it on the compromised system. There are certain features of a malware incident that are sufficiently distinctive to warrant a broad search of the system for related information. Such distinctive items include:
openvpn
, vncviewer
).ac 10
9d 88
) both in little and big endian formats. Therefore, it might be necessary to construct multiple keywords for a single IP address.# blkls -A /evidence/phalanx2.dd | strings –t d | grep “Nov
13”
FIGURE 3.19 - Salvaging deleted log entries dated Nov 13 by searching for strings in unallocated space that is extracted from a forensic duplicate using the blkls utility from The Sleuth Kit
The use of partitions in Linux to group different types of data can make keyword searching more effective. For instance, rather than scouring the entire hard drive, digital investigators may be able to recover all deleted log entries by simply searching the partition that contains log files.
About the authors:
Cameron H. Malin is Special Agent with the Federal Bureau of Investigation assigned to a Cyber Crime squad in Los Angeles, California, where he is responsible for the investigation of computer intrusion and malicious code matters. Special Agent Malin is the founder and developer of the FBI’s Technical Working Group on Malware Analysis and Incident Response. Special Agent Malin is a Certified Ethical Hacker (C|EH) as designated by the International Council of E-Commerce Consultants, a Certified Information Systems Security Professional (CISSP), as designated by the International Information Systems Security Consortium, a GIAC certified Reverse-Engineering Malware Professional (GREM), GIAC Certified Intrusion Analyst (GCIA), GIAC Certified Incident Handler (GCIH), and a GIAC Certified Forensic Analyst (GCFA), as designated by the SANS Institute.
Eoghan Casey is an internationally recognized expert in data breach investigations and information security forensics. He is founding partner of CASEITE.com, and co-manages the Risk Prevention and Response business unit at DFLabs. Over the past decade, he has consulted with many attorneys, agencies, and police departments in the United States, South America, and Europe on a wide range of digital investigations, including fraud, violent crimes, identity theft, and on-line criminal activity. Eoghan has helped organizations investigate and manage security breaches, including network intrusions with international scope. He has delivered expert testimony in civil and criminal cases, and has submitted expert reports and prepared trial exhibits for computer forensic and cyber-crime cases. In addition to his casework and writing the foundational book Digital Evidence and Computer Crime, Eoghan has worked as R&D Team Lead in the Defense Cyber Crime Institute (DCCI) at the Department of Defense Cyber Crime Center (DC3) helping enhance their operational capabilities and develop new techniques and tools. He also teaches graduate students at Johns Hopkins University Information Security Institute and created the Mobile Device Forensics course taught worldwide through the SANS Institute. He has delivered keynotes and taught workshops around the globe on various topics related to data breach investigation, digital forensics and cyber security. Eoghan has performed thousands of forensic acquisitions and examinations, including Windows and UNIX systems, Enterprise servers, smart phones, cell phones, network logs, backup tapes, and database systems. He also has information security experience, as an Information Security Officer at Yale University and in subsequent consulting work. He has performed vulnerability assessments, deployed and maintained intrusion detection systems, firewalls and public key infrastructures, and developed policies, procedures, and educational programs for a variety of organizations. Eoghan has authored advanced technical books in his areas of expertise that are used by practitioners and universities around the world, and he is Editor-in-Chief of Elsevier's International Journal of Digital Investigation.
James M. Aquilina, Esq. is the Managing Director and Deputy General Counsel of Stroz Friedberg, LLC, a consulting and technical services firm specializing in computer forensics; cyber-crime response; private investigations; and the preservation, analysis and production of electronic data from single hard drives to complex corporate networks. As the head of the Los Angeles Office, Mr. Aquilina supervises and conducts digital forensics and cyber-crime investigations and oversees large digital evidence projects. Mr. Aquilina also consults on the technical and strategic aspects of anti-piracy, antispyware, and digital rights management (DRM) initiatives for the media and entertainment industries, providing strategic thinking, software assurance, testing of beta products, investigative assistance, and advice on whether the technical components of the initiatives implicate the Computer Fraud and Abuse Act and anti-spyware and consumer fraud legislation. His deep knowledge of botnets, distributed denial of service attacks, and other automated cyber-intrusions enables him to provide companies with advice to bolster their infrastructure protection.
29 Sep 2014