Forensic analysis

Mar 12, 2020

Post-compromise analysis on Red-Hat-like operating systems

This describes some useful techniques for performing a post-compromise forensic analysis of a Red-Hat-like operating system, such as Red Hat Enterprise Linux, CentOS, and Fedora. While these instructions are RPM-centric, similar techniques could apply elsewhere, for example by replacing rpm with dpkg and yum with apt. We assume that you have an image of the compromised host’s disk partition (or partitions) which we refer to here as /evidence/partition.img. Where a whole disk is required, we use the name /evidence/disk.img. (See below for how to manipulate a partition within a whole-disk image and how to deal with Linux Volume Management.) We further assume that the compromised host was protected by neither a trusted boot process, nor a file integrity tool such as Tripwire, nor an encrypted disk. Such protections would only make this task easier.

Partition carving and Linux Volume Management

Given a whole-disk image, you can mount a single partition by first identifying where the partition begins with parted and then providing this byte offset to the mount command. The following example assumes a 512-byte sector size while mounting partition one:

$ parted /evidence/disk.img unit s print
WARNING: You are not superuser.  Watch out for permissions.
Model:  (file)
Disk /evidence/disk.img: 62914560s
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start     End        Size       Type     File system  Flags
 1      2048s     1026047s   1024000s   primary  xfs          boot
 2      1026048s  62914559s  61888512s  primary               lvm

$ sudo mount -o offset=$((2048*512)) /evidence/disk.img /mnt/

In the above example, the second partition is dedicated to LVM. You can mount partitions subject to LVM in this way:

$ sudo kpartx -va /evidence/disk.img
add map loop1p1 (253:4): 0 1024000 linear /dev/loop1 2048
add map loop1p2 (253:5): 0 61888512 linear /dev/loop1 1026048
$ ls /dev/mapper/
centos-root  loop1p1
centos-swap  loop1p2
control
$ sudo mount /dev/mapper/centos-root /mnt

Analysis setup

First, you will mount /evidence/partition.img into the filesystem of a trustworthy (i.e., likely uncompromised) computer. It would not hurt to isolate this computer from the rest of your network, but it is very important that you not immediately directly boot /evidence/partition.img. (Some analysis will require that you boot partition.img; you will need to decide when to do so in order to perform tests like netstat checks.)

mount /evidence/partition.img /mnt

It is possible that you will start with a disk image which is in some virtualization-platform format. If this is the case, then you can use qemu-img to convert it to a raw disk image. For an example of a disk-image-format conversion, see our notes on virtualization.

Weak RPM verification

The following will check the integrity of the files which were installed on /evidence/partition.img (mounted at /mnt) from RPM packages. However, this check relies on the integrity of the (possibly compromised) RPM database on /evidence/partition.img, and so the results here cannot be fully trusted. Despite this step’s flaws, it might turn up something of interest, especially if the attacker was careless.

sudo rpm --root /mnt --verify -a | grep "\(^[^.]\)\|^.[^.]\|^..[^.]\|^...[^.]\|^....[^.]\|^.....[^.]\|^......[^.]\|^.......[^.T]\|^........[^.]"

The rpm manpage documents the details of the output from this command. In summary, it indicates which files have changed since being installed from an RPM package. The use of grep prints only the files RPM thinks are modified other than mere mtime changes.

Trustworthy RPM verification

Here we describe how to check the integrity of a program named foo which we assume to have been installed using the package foo-version.rpm. You can identify version by running sudo rpm --root /mnt -q foo. Let us first install a trustworthy copy of foo on our trustworthy host:

sudo yum --releasever=/ --installroot=/tmp/scratch install foo-version

You will likely find that yum installs a number of dependencies for foo. This can cause trouble, because we are about to compare files on the compromised host with the files we download here. Rather than allow yum to install the latest versions of dependencies, you ought to install the same version which is present on the compromised host. Note the dependencies, identify the proper versions using rpm ... -q ..., and explicitly add these dependencies to the yum command line. You should write a script to perform these steps.

Once you have installed a trustworthy copy of the software which exists on the compromised host, you can start comparing files. This is a matter of running

md5sum /mnt/path/to/file

and

md5sum /scratch/path/to/file

and comparing the hashes. Again, you should write a script to automate these steps.

Considering the nature of dynamically-linked programs

Does a proper hash of /usr/bin/foo mean that foo is not compromised? No! Modern systems employ shared libraries, which a dynamic linker combines with program code at runtime. To illustrate this, run the following command to display the shared libraries that foo makes use of:

ldd /usr/bin/foo

If you run this command, then you should see that foo makes use of a number of shared libraries. You must check the integrity of each of these in addition to foo itself. Some programs are instead statically linked (i.e., all of their code is present in a single executable file). The file command will report whether a given executable is dynamically or statically linked.

Some environment variables influence from where the dynamic linker will load shared libraries. These include LD_LIBRARY_PATH and LD_PRELOAD. An attacker could set these variables to cause the linker to load a library from an unexpected location. Likewise, the dynamic linker honors a number of configuration files, including /etc/ld.so.conf and /etc/ld.so.preload.

Particularly dangerous programs

Setuid-bit programs run with the privileges of the owner of the program's file within the filesystem instead of the parent process. This mechanism is often used to temporarily escalate privileges, and so compromised setuid-bit programs are particularly dangerous. You can list the setuid-bit programs on the compromised system by running the following command:

find /mnt -perm -4000 -type f

A second set of commands which are dangerous are those that listen on a network socket. If they are compromised, then they might give a remote attacker access to the computer. To view the programs which are listening on (or are connected via) UNIX-domain, TCP, and UDP sockets, run:

netstat -ap

It is wise to compare these setuid and network-facing programs to the list of files modified since being installed by RPM.

Even more dangerous is a compromised kernel or bootloader, which we describe next.

The boot process

Modern PC hardware first executes a bootloader either from a master-boot record or—on capable firmware—from a particular file within a filesystem. In either case, the bootloader could be compromised. To investigate the 512-byte master-boot record, copy it to /scratch using:

dd bs=1 count=512 if=/evidence/disk.img of=/scratch/mbr

Other documentation describes the structure of the master-boot record.

The GRUB 2 bootloader places a stage-one bootloader, boot.img, within the master-boot record. This stage-one loader loads a stage-1.5 loader, core.img, which exists between the master-boot record and the first disk partition. The stage 1.5 loader also maintains a configuration file and a number of loadable modules which perform various tasks such as parsing various filesystem layouts. GRUB 2's first interaction with a filesystem takes place while executing its stage-two loader which finally loads the OS kernel. The stage-two loader is generally found in the /boot directory, and its configuration exists at /boot/grub2/config AKA /etc/grub2.cfg.

Checking GRUB 2 requires inspecting the stage-one, stage-1.5, and stage-two loaders as well as the related configuration files and loadable modules.

Instead of verifying each of GRUB 2's installed files, you can merely overwrite them with trustworthy copies. Updating a GRUB 2 installation is a matter of running as root:

grub2-mkconfig > /boot/grub2/grub.cfg
grub2-install /dev/sda

Kernels exist in /boot, and are named vmlinuz-version. Each kernel can load dynamic modules into its address space, and these modules exist in /lib/modules.

Checking a kernel is a matter of checking the kernel itself, as well as each of its loadable modules. While you can use the lsmod to review the currently loaded kernel modules, it is likely that lsmod is compromised.

Accounts and the login process

The files /etc/passwd, /etc/shadow, and /etc/group define standard UNIX accounts and groups. However, many programs perform authentication using Pluggable Authentication Modules (PAM) which provide many other authentication techniques. Such techniques include Kerberos, NIS, and so on. The PAM modules each PAM-capable program uses to authenticate are defined using configuration files in /etc/pam.d. Thus checking for proper authentication involves:

  • Checking the integrity of each authenticating program
  • Checking the PAM configuration of each authenticating program
  • Checking the accounts themselves (e.g., expected accounts with strong passwords)

Graphical login authentication is generally performed using a display manager such as gdm, and text logins usually use agetty which spawns login.

Access controls

Access controls constrain programs. On UNIX, OS objects—such as files—bear permissions. For example, ls -l /path/to/file will display the permissions (along with other information) for the file at /path/to/file. An attacker might perturb these permissions to make returning to the compromised computer easier in the future. Thus it is wise to check the permissions of installed files in a manner similar to how you checked the integrity of the files themselves.

Permissions on block and 8char device nodes* are particularly troublesome. For example, if a block device which corresponds to a disk has permissive access controls, then an attacker could use this block device to undermine the access controls within the filesystem contained therein. To enumerate all of the device nodes on the compromised host, run

find /mnt -type b -o -type c

Other forms of access controls provide stronger or more expressive protections. For example, a filesystem’s access-control lists can be displayed using getfacl /path/to/file. SELinux policy source and binary files exist in /etc/selinux/, and SELinux can be activated/deactivated by editing /etc/sysconfig/selinux.

Checklist

  1. Verify kernel, kernel modules, and GRUB 2 (do not forget to look for extra kernel modules)
  2. Verify setuid programs and their shared libraries
  3. Verify programs and their shared libraries
  4. Verify permissions (especially device nodes)
  5. Verify SELinux policy
  6. Review PAM configurations
  7. Review accounts
  8. Review boot services
  9. Review cron and at jobs