Floating Octothorpe

Linux teardown checklist

With virtualisation it's very easy to quickly spin up new Linux hosts for development. Unfortunately this also means it's very easy to have a large number of unused hosts and old VM images. This post is going to go through places to check before completely deleting a Linux host.

Shell history

Assuming the host is headless, shell history is one of the best places to look for information on what a host was used for. The history shell built-in can be used to show recently run commands:

$ history
1  shutdown -h now
2  python
3  cat /etc/yum.repos.d/CentOS-Base.repo
4  vim /etc/yum.repos.d/CentOS-Base.repo
5  yum search python-cv
6  yum search opencv
7  yum install opencv-python

Note: if multiple users have been set up, make sure you check the history for each user account.

Services and listening ports

ss can be used to quickly identify listening services which might have associated data you want to backup:

$ ss --numeric --tcp --udp --listening --process
Netid State      Recv-Q Send-Q  Local Address:Port   Peer Address:Port
udp   UNCONN     0      0                   *:38680             *:*      users:(("dhclient",pid=649,fd=20))
udp   UNCONN     0      0                   *:68                *:*      users:(("dhclient",pid=649,fd=6))
udp   UNCONN     0      0                  :::38602            :::*      users:(("dhclient",pid=649,fd=21))
tcp   LISTEN     0      128                 *:22                *:*      users:(("sshd",pid=927,fd=3))
tcp   LISTEN     0      100                *:*      users:(("master",pid=1023,fd=13))
tcp   LISTEN     0      128                :::22               :::*      users:(("sshd",pid=927,fd=4))
tcp   LISTEN     0      100               ::1:25               :::*      users:(("master",pid=1023,fd=14))

On systems using systemd, systemctl can also be used to identify enabled services:

$ systemctl list-unit-files --state enabled
UNIT FILE                                   STATE
accounts-daemon.service                     enabled
auditd.service                              enabled
bluetooth.service                           enabled
crond.service                               enabled
dbus-org.bluez.service                      enabled
dbus-org.freedesktop.NetworkManager.service enabled
dbus-org.freedesktop.nm-dispatcher.service  enabled
display-manager.service                     enabled

System configuration

System configuration can normally be found in /etc/. The exact files you want to backup will obviously depend on the system, however below are a few locations you may want to check for configuration:

User accounts and home directories

Home directories are an obvious place to check for files which you might want to recover before deleting a system. Initially just listing the directories in /home is a good way to work out where you need to check:

$ ls /home/

It's also worth checking entries in /etc/passwd for any non-standard home directory locations.

Once you've got a list of user home directories, have a look through each directory. When looking through the files, make sure you don't forget to look at hidden files. Below are a few hidden files you might want to consider backing up:


Besides users home directories, the following places may contain data that might be worth backing up:

Package manager

Before removing a system, it's often useful making a list of packages which were installed. This can be helpful when trying to locate configuration you might want to backup, or if you decide to re-create the system.

Yum and RPM

For CentOS systems yum history can be used to review Yum transactions:

$ yum history
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
ID     | Login user               | Date and time    | Action(s)      | Altered
     7 | root <root>              | 2016-12-11 19:46 | Install        |    4
     6 | root <root>              | 2016-12-11 19:32 | Install        |    1
     5 | root <root>              | 2016-12-11 19:28 | Install        |  194
     4 | root <root>              | 2016-12-11 19:04 | Install        |    1
     3 | root <root>              | 2016-12-11 18:28 | I, U           |  178 EE
     2 | root <root>              | 2016-12-11 18:24 | Install        |   63
     1 | System <unset>           | 2016-09-29 14:06 | Install        |  312

You can also use yum history info to get more information on a specific transaction:

$ yum history info 4
Loaded plugins: fastestmirror
Repodata is over 2 weeks old. Install yum-cron? Or run: yum makecache fast
Transaction ID : 4
Begin time     : Sun Dec 11 19:04:06 2016
Begin rpmdb    : 378:21fccfb42d3b1e74979111e732852b5061a2c6ed
End time       :            19:04:07 2016 (1 seconds)
End rpmdb      : 379:75a7237da62e0b9a892be970216af98ad488fb76
User           : root <root>
Return-Code    : Success
Command Line   : install https://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm
Transaction performed with:
    Installed     rpm-4.11.3-17.el7.x86_64                      @base-local
    Installed     yum-3.4.3-132.el7.centos.0.1.noarch           @base-local
    Installed     yum-plugin-fastestmirror-1.1.31-34.el7.noarch @base-local
Packages Altered:
    Install epel-release-7-8.noarch @/epel-release-7-8.noarch
history info

Yum will also log transaction information to /var/log/yum.log:

$ grep epel /var/log/yum.log
Dec 11 19:04:06 Installed: epel-release-7-8.noarch

For more information on using yum refer to the Yum man page.

You may also want to produce a list of installed packages, this can be done with the rpm command:

$ rpm -qa | sort

As well as listing packages, the rpm command can also be used to find modified files. This is useful for identifying configuration that may have been customised:

$ rpm -Va
.......T.    /lib/modules/3.10.0-229.el7.x86_64/modules.devname
.......T.    /lib/modules/3.10.0-229.el7.x86_64/modules.softdep
S.5....T.  c /etc/yum.repos.d/CentOS-Base.repo
missing     /run/gdm
.M....G..    /var/log/gdm
....L....  c /etc/pam.d/fingerprint-auth
....L....  c /etc/pam.d/password-auth
....L....  c /etc/pam.d/postlogin
....L....  c /etc/pam.d/smartcard-auth
....L....  c /etc/pam.d/system-auth
missing     /var/run/wpa_supplicant
S.5....T.  c /etc/plymouth/plymouthd.conf
S.5....T.  c /etc/sysconfig/authconfig

The characters at the start of each line denote the following:

Apt and dpkg

Debian based systems like Ubuntu use the Apt package management system. Unfortunately Apt doesn't have a direct equivalent to yum history, however the following log files should provide details on when packages were installed, updated, and removed:

To produce a list of installed Debian packages the dpkg command can be used:

$ dpkg -l
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                          Version                        Architecture Description
ii  acl                           2.2.52-3+b1                    amd64        Access control list utilities
ii  adduser                       3.115                          all          add and remove users and groups
ii  apache2                       2.4.25-3+deb9u3                amd64        Apache HTTP Server
ii  apache2-bin                   2.4.25-3+deb9u3                amd64        Apache HTTP Server (modules and other binary files)
ii  apache2-data                  2.4.25-3+deb9u3                all          Apache HTTP Server (common files)
ii  apache2-utils                 2.4.25-3+deb9u3                amd64        Apache HTTP Server (utility programs for web servers)
ii  apt                           1.4.7                          amd64        commandline package manager
ii  base-files                    9.9+deb9u1                     amd64        Debian base system miscellaneous files

Cron Jobs

Have a quick look in the following directories for any cron jobs you might have set up:

SSH keys and config

As well as checking in ~/.ssh for each user, you may also want to backup the host's SSH keys. If you're rebuilding a host these can be reused to keep the same host fingerprint. This will prevent conflicting known_hosts entries when connecting to the system. The public and private keys can normally be found in /etc/ssh/:

$ ls -l /etc/ssh/ssh_host*
-rw-r-----. 1 root ssh_keys  227 Sep 29  2016 /etc/ssh/ssh_host_ecdsa_key
-rw-r--r--. 1 root root      162 Sep 29  2016 /etc/ssh/ssh_host_ecdsa_key.pub
-rw-r-----. 1 root ssh_keys  387 Sep 29  2016 /etc/ssh/ssh_host_ed25519_key
-rw-r--r--. 1 root root       82 Sep 29  2016 /etc/ssh/ssh_host_ed25519_key.pub
-rw-r-----. 1 root ssh_keys 1679 Sep 29  2016 /etc/ssh/ssh_host_rsa_key
-rw-r--r--. 1 root root      382 Sep 29  2016 /etc/ssh/ssh_host_rsa_key.pub