Linux teardown checklist

15 December 2017

With virtualisation it's very easy to quickly spin up new Linux hosts for development. Unfortunately this also means it's very easy to have a large number of unused hosts and old VM images. This post is going to go through places to check before completely deleting a Linux host.

Shell history

Assuming the host is headless, shell history is one of the best places to look for information on what a host was used for. The history shell built-in can be used to show recently run commands:

$ history
1  shutdown -h now
2  python
3  cat /etc/yum.repos.d/CentOS-Base.repo
4  vim /etc/yum.repos.d/CentOS-Base.repo
5  yum search python-cv
6  yum search opencv
7  yum install opencv-python
...

Note: if multiple users have been set up, make sure you check the history for each user account.

Services and listening ports

ss can be used to quickly identify listening services which might have associated data you want to backup:

$ ss --numeric --tcp --udp --listening --process
Netid State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port
udp   UNCONN     0      0                   *:38680             *:*      users:(("dhclient",pid=649,fd=20))
udp   UNCONN     0      0                   *:68                *:*      users:(("dhclient",pid=649,fd=6))
udp   UNCONN     0      0                  :::38602            :::*      users:(("dhclient",pid=649,fd=21))
tcp   LISTEN     0      128                 *:22                *:*      users:(("sshd",pid=927,fd=3))
tcp   LISTEN     0      100         127.0.0.1:25                *:*      users:(("master",pid=1023,fd=13))
tcp   LISTEN     0      128                :::22               :::*      users:(("sshd",pid=927,fd=4))
tcp   LISTEN     0      100               ::1:25               :::*      users:(("master",pid=1023,fd=14))

On systems using systemd, systemctl can also be used to identify enabled services:

$ systemctl list-unit-files --state enabled
UNIT FILE                                   STATE
accounts-daemon.service                     enabled
auditd.service                              enabled
bluetooth.service                           enabled
crond.service                               enabled
dbus-org.bluez.service                      enabled
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.nm-dispatcher.service  enabled
display-manager.service                     enabled
...

System configuration

System configuration can normally be found in /etc/. The exact files you want to backup will obviously depend on the system, however below are a few locations you may want to check for configuration:

/etc/fstab: file system mount points.
/etc/network/: on Debian systems network configuration is normally kept here.
/etc/sysconfig/network and /etc/sysconfig/network-scripts/: on Red Hat based systems network configuration is normally kept here.
/etc/hosts: static host entries.
/etc/resolv.conf: DNS configuration.
/etc/sysconfig/iptables: on Red Hat base systems, persistent iptables rules are stored here. You may also want to run iptables-save to dump the rules from memory.
/etc/sudoers and /etc/sudoers.d/: sudo rules.
/etc/profile and /etc/profile.d/: global shell login scripts.
/etc/apt/sources.list and /etc/apt/sources.list.d/: repository configuration for Apt on Debian based systems.
/etc/yum.conf and /etc/yum.repos.d/: Yum configuration on Red Hat based systems.

User accounts and home directories

Home directories are an obvious place to check for files which you might want to recover before deleting a system. Initially just listing the directories in /home is a good way to work out where you need to check:

$ ls /home/
alice

It's also worth checking entries in /etc/passwd for any non-standard home directory locations.

Once you've got a list of user home directories, have a look through each directory. When looking through the files, make sure you don't forget to look at hidden files. Below are a few hidden files you might want to consider backing up:

.ssh/: SSH configuration including private keys is normally stored in ~/.ssh. It's also worth making a note of any accounts using ~/.ssh/authorized_keys to allow remote users to connect.
.bashrc and .bash_profile: If you've setup any useful custom aliases you might want to back them up. Bash configuration may also point you at software being used on the system. For example if PGDATA is being set, there is a good chance there may be a PostgreSQL database you might need to back up.
.vimrc and .vim/: If you use Vim, check for useful configuration you might want to carry over to another system.
.gnupg/: Check the GPG configuration directory for any keys you might want to keep.

Data

Besides users home directories, the following places may contain data that might be worth backing up:

/var/lib/: this directory is used to hold variable state information. It's often used by services to store data, for example PostgreSQL often stores database files under /var/lib/pgsql/data/.
/srv/: data for services is sometimes kept here.
/var/www/html: This is the default document root for Apache.
/var/spool/mail/: user emails are normally stored here.
/var/log: you many want to check system logs for useful info.

Package manager

Before removing a system, it's often useful making a list of packages which were installed. This can be helpful when trying to locate configuration you might want to backup, or if you decide to re-create the system.

Yum and RPM

For CentOS systems yum history can be used to review Yum transactions:

$ yum history
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
ID     | Login user               | Date and time    | Action(s)      | Altered
------------------------------------------------------------------------------- 
     7 | root <root>              | 2016-12-11 19:46 | Install        |    4
     6 | root <root>              | 2016-12-11 19:32 | Install        |    1
     5 | root <root>              | 2016-12-11 19:28 | Install        |  194
     4 | root <root>              | 2016-12-11 19:04 | Install        |    1
     3 | root <root>              | 2016-12-11 18:28 | I, U           |  178 EE
     2 | root <root>              | 2016-12-11 18:24 | Install        |   63
     1 | System <unset>           | 2016-09-29 14:06 | Install        |  312

You can also use yum history info to get more information on a specific transaction:

$ yum history info 4
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Transaction ID : 4
Begin time     : Sun Dec 11 19:04:06 2016
Begin rpmdb    : 378:21fccfb42d3b1e74979111e732852b5061a2c6ed
End time       :            19:04:07 2016 (1 seconds)
End rpmdb      : 379:75a7237da62e0b9a892be970216af98ad488fb76
User           : root <root>
Return-Code    : Success
Command Line   : install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
Transaction performed with:
    Installed     rpm-4.11.3-17.el7.x86_64                      @base-local
    Installed     yum-3.4.3-132.el7.centos.0.1.noarch           @base-local
    Installed     yum-plugin-fastestmirror-1.1.31-34.el7.noarch @base-local
Packages Altered:
    Install epel-release-7-8.noarch @/epel-release-7-8.noarch
history info

Yum will also log transaction information to /var/log/yum.log:

$ grep epel /var/log/yum.log
Dec 11 19:04:06 Installed: epel-release-7-8.noarch

For more information on using yum refer to the Yum man page.

You may also want to produce a list of installed packages, this can be done with the rpm command:

$ rpm -qa | sort
abattis-cantarell-fonts-0.0.16-3.el7.noarch
accountsservice-0.6.35-9.el7.x86_64
accountsservice-libs-0.6.35-9.el7.x86_64
acl-2.2.51-12.el7.x86_64
adwaita-cursor-theme-3.14.1-1.el7.noarch
adwaita-gtk2-theme-3.14.2.2-2.el7.x86_64
adwaita-icon-theme-3.14.1-1.el7.noarch
aic94xx-firmware-30-6.el7.noarch
...

As well as listing packages, the rpm command can also be used to find modified files. This is useful for identifying configuration that may have been customised:

$ rpm -Va
.......T.    /lib/modules/3.10.0-229.el7.x86_64/modules.devname
.......T.    /lib/modules/3.10.0-229.el7.x86_64/modules.softdep
S.5....T.  c /etc/yum.repos.d/CentOS-Base.repo
missing     /run/gdm
.M....G..    /var/log/gdm
....L....  c /etc/pam.d/fingerprint-auth
....L....  c /etc/pam.d/password-auth
....L....  c /etc/pam.d/postlogin
....L....  c /etc/pam.d/smartcard-auth
....L....  c /etc/pam.d/system-auth
missing     /var/run/wpa_supplicant
S.5....T.  c /etc/plymouth/plymouthd.conf
S.5....T.  c /etc/sysconfig/authconfig
...

The characters at the start of each line denote the following:

S: file Size differs
M: Mode differs (includes permissions and file type)
5: digest (formerly MD5 sum) differs
D: Device major/minor number mismatch
L: readLink(2) path mismatch
U: User ownership differs
G: Group ownership differs
T: mTime differs
P: caPabilities differ

Apt and dpkg

Debian based systems like Ubuntu use the Apt package management system. Unfortunately Apt doesn't have a direct equivalent to yum history, however the following log files should provide details on when packages were installed, updated, and removed:

/var/log/apt/history.log
/var/log/dpkg.log

To produce a list of installed Debian packages the dpkg command can be used:

$ dpkg -l
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version                        Architecture Description
+++-=============================-==============================-============-===============================================================================
ii  acl                           2.2.52-3+b1                    amd64        Access control list utilities
ii  adduser                       3.115                          all          add and remove users and groups
ii  apache2                       2.4.25-3+deb9u3                amd64        Apache HTTP Server
ii  apache2-bin                   2.4.25-3+deb9u3                amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                  2.4.25-3+deb9u3                all          Apache HTTP Server (common files)
ii  apache2-utils                 2.4.25-3+deb9u3                amd64        Apache HTTP Server (utility programs for web servers)
ii  apt                           1.4.7                          amd64        commandline package manager
ii  base-files                    9.9+deb9u1                     amd64        Debian base system miscellaneous files
...

Cron Jobs

Have a quick look in the following directories for any cron jobs you might have set up:

/var/spool/cron/
/etc/cron.daily/
/etc/cron.hourly/
/etc/cron.weekly/
/etc/cron.monthly/
/etc/cron.d/

SSH keys and config

As well as checking in ~/.ssh for each user, you may also want to backup the host's SSH keys. If you're rebuilding a host these can be reused to keep the same host fingerprint. This will prevent conflicting known_hosts entries when connecting to the system. The public and private keys can normally be found in /etc/ssh/:

$ ls -l /etc/ssh/ssh_host*
-rw-r-----. 1 root ssh_keys  227 Sep 29  2016 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r--. 1 root root      162 Sep 29  2016 /etc/ssh/ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys  387 Sep 29  2016 /etc/ssh/ssh_host_ed25519_key
-rw-r--r--. 1 root root       82 Sep 29  2016 /etc/ssh/ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys 1679 Sep 29  2016 /etc/ssh/ssh_host_rsa_key
-rw-r--r--. 1 root root      382 Sep 29  2016 /etc/ssh/ssh_host_rsa_key.pub