Post-compromise analysis on Red-Hat-like operating systems
This describes some useful techniques for performing a post-compromise forensic analysis of a Red-Hat-like operating system, such as Red Hat Enterprise Linux, CentOS, and Fedora. While these instructions are RPM-centric, similar techniques could apply elsewhere, for example by replacing rpm with dpkg and yum with apt. We assume that you have an image of the compromised host's disk partition (or partitions) which we refer to here as /evidence/partition.img. Where a whole disk is required, we use the name /evidence/disk.img. (See below for how to manipulate a partition within a whole-disk image and how to deal with Linux Volume Management.) We further assume that the compromised host was protected by neither a trusted boot process, nor a file integrity tool such as Tripwire, nor an encrypted disk. Such protections would only make this task easier.
Partition carving and Linux Volume ManagementGiven a whole-disk image, you can mount a single partition by first identifying where the partition begins with parted and then providing this byte offset to the mount command. The following example assumes a 512-byte sector size while mounting partition one:
$ parted /evidence/disk.img unit s print WARNING: You are not superuser. Watch out for permissions. Model: (file) Disk /evidence/disk.img: 62914560s Sector size (logical/physical): 512B/512B Partition Table: msdos Disk Flags:In the above example, the second partition is dedicated to LVM. You can mount partitions subject to LVM in this way:
Number Start End Size Type File system Flags 1 2048s 1026047s 1024000s primary xfs boot 2 1026048s 62914559s 61888512s primary lvm
$ sudo mount -o offset=$((2048*512)) /evidence/disk.img /mnt/
$ sudo kpartx -va /evidence/disk.img add map loop1p1 (253:4): 0 1024000 linear /dev/loop1 2048 add map loop1p2 (253:5): 0 61888512 linear /dev/loop1 1026048 $ ls /dev/mapper/ centos-root loop1p1 centos-swap loop1p2 control $ sudo mount /dev/mapper/centos-root /mnt
First, you will mount /evidence/partition.img into the filesystem of a trustworthy (i.e., likely uncompromised) computer. It would not hurt to isolate this computer from the rest of your network, but it is very important that you not immediately directly boot /evidence/partition.img. (Some analysis will require that you boot partition.img; you will need to decide when to do so in order to perform tests like netstat checks.)
mount /evidence/partition.img /mnt
It is possible that you will start with a disk image which is in some virtualization-platform format. If this is the case, then you can use qemu-img to convert it to a raw disk image. For an example of a disk-image-format conversion, see our notes on virtualization.
Weak RPM verification
The following will check the integrity of the files which were installed on /evidence/partition.img (mounted at /mnt) from RPM packages. However, this check relies on the integrity of the (possibly compromised) RPM database on /evidence/partition.img, and so the results here cannot be fully trusted. Despite this step's flaws, it might turn up something of interest, especially if the attacker was careless.
sudo rpm --root /mnt --verify -a | grep "\(^[^.]\)\|^.[^.]\|^..[^.]\|^...[^.]\|^....[^.]\|^.....[^.]\|^......[^.]\|^.......[^.T]\|^........[^.]"
The rpm manpage documents the details of the output from this command. In summary, it indicates which files have changed since being installed from an RPM package. The use of grep prints only the files RPM thinks are modified other than mere mtime changes.
Trustworthy RPM verification
Here we describe how to check the integrity of a program named foo which we assume to have been installed using the package foo-version.rpm. You can identify version by running sudo rpm --root /mnt -q foo. Let us first install a trustworthy copy of foo on our trustworthy host:
sudo yum --releasever=/ --installroot=/tmp/scratch install foo-version
You will likely find that yum installs a number of dependencies for foo. This can cause trouble, because we are about to compare files on the compromised host with the files we download here. Rather than allow yum to install the latest versions of dependencies, you ought to install the same version which is present on the compromised host. Note the dependencies, identify the proper versions using rpm … -q …, and explicitly add these dependencies to the yum command line. You should write a script to perform these steps.
Once you have installed a trustworthy copy of the software which exists on the compromised host, you can start comparing files. This is a matter of running
and comparing the hashes. Again, you should write a script to automate these steps.
Considering the nature of dynamically-linked programs
Does a proper hash of /usr/bin/foo mean that foo is not compromised? No! Modern systems employ shared libraries, which a dynamic linker combines with program code at runtime. To illustrate this, run the following command to display the shared libraries which foo makes use of:
If you run this command, then you should see that foo makes use of a number of shared libraries. You must check the integrity of each of these in addition to foo itself. Some programs are instead statically linked (i.e., all of their code is present in a single executable file). The file command will report whether a given executable is dynamically or statically linked.
Some environment variables influence from where the dynamic linker will load shared libraries. These include LD_LIBRARY_PATH and LD_PRELOAD. An attacker could set these variables to cause the linker to load a library from an unexpected location. Likewise, the dynamic linker honors a number of configuration files, including /etc/ld.so.conf and /etc/ld.so.preload.
Particularly dangerous programs
Setuid-bit programs run with the privileges of the owner of the program's file within the filesystem instead of the parent process. This mechanism is often used to temporarily escalate privileges, and so compromised setuid-bit programs are particularly dangerous. You can list the setuid-bit programs on the compromised system by running the following command:
find /mnt -perm -4000 -type f
A second set of commands which are dangerous are those that listen on a network socket. If they are compromised, then they might give a remote attacker access to the computer. To view the programs which are listening on (or are connected via) UNIX-domain, TCP, and UDP sockets, run:
It is wise to compare these setuid and network-facing programs to the list of files modified since being installed by RPM.
Even more dangerous is a compromised kernel or bootloader, which we describe next.
The boot process
Modern PC hardware first executes a bootloader either from a master-boot record or—on capable firmware—from a particular file within a filesystem. In either case, the bootloader could be compromised. To investigate the 512-byte master-boot record, copy it to /scratch using:
dd bs=1 count=512 if=/evidence/disk.img of=/scratch/mbr
Other documentation describes the structure of the master-boot record.
The GRUB 2 bootloader places a stage-one bootloader, boot.img, within the master-boot record. This stage-one loader loads a stage-1.5 loader, core.img, which exists between the master-boot record and the first disk partition. The stage 1.5 loader also maintains a configuration file and a number of loadable modules which perform various tasks such as parsing various filesystem layouts. GRUB 2's first interaction with a filesystem takes place while executing its stage-two loader which finally loads the OS kernel. The stage-two loader is generally found in the /boot directory, and its configuration exists at /boot/grub2/config AKA /etc/grub2.cfg.
Checking GRUB 2 requires inspecting the stage-one, stage-1.5, and stage-two loaders as well as the related configuration files and loadable modules.
Instead of verifying each of GRUB 2's installed files, you can merely overwrite them with trustworthy copies. Updating a GRUB 2 installation is a matter of running as root:
grub2-mkconfig > /boot/grub2/grub.cfg grub2-install /dev/sda
Kernels exist in /boot, and are named vmlinuz-version. Each kernel can load dynamic modules into its address space, and these modules exist in /lib/modules.
Checking a kernel is a matter of checking the kernel itself, as well as each of its loadable modules. While you can use the lsmod to review the currently loaded kernel modules, it is likely that lsmod is compromised.
Accounts and the login process
The files /etc/passwd, /etc/shadow, and /etc/group define standard UNIX accounts and groups. However, many programs perform authentication using Pluggable Authentication Modules (PAM) which provide many other authentication techniques. Such techniques include Kerberos, NIS, and so on. The PAM modules each PAM-capable program uses to authenticate are defined using configuration files in /etc/pam.d. Thus checking for proper authentication involves:
- Checking the integrity of each authenticating program
- Checking the PAM configuration of each authenticating program
- Checking the accounts themselves (e.g., expected accounts with strong passwords)
Access controls constrain programs. On UNIX, OS objects—such as files—bear permissions. For example, ls -l /path/to/file will display the permissions (along with other information) for the file at /path/to/file. An attacker might perturb these permissions to make returning to the compromised computer easier in the future. Thus it is wise to check the permissions of installed files in a manner similar to how you checked the integrity of the files themselves.
Permissions on block and char device nodes are particularly troublesome. For example, if a block device which corresponds to a disk has permissive access controls, then an attacker could use this block device to undermine the access controls within the filesystem contained therein. To enumerate all of the device nodes on the compromised host, run
find /mnt -type b -o -type c
Other forms of access controls provide stronger or more expressive protections. For example, a filesystem's access-control lists can be displayed using getfacl /path/to/file. SELinux policy source and binary files exist in /etc/selinux/, and SELinux can be activated/deactivated by editing /etc/sysconfig/selinux.
- Verify kernel, kernel modules, and GRUB 2 (do not forget to look for extra kernel modules)
- Verify setuid programs and their shared libraries
- Verify programs and their shared libraries
- Verify permissions (especially device nodes)
- Verify SELinux policy
- Review PAM configurations
- Review accounts
- Review boot services
- Review cron and at jobs