Remote logging with Rsyslog

17 February 2017

Rsyslog is one of the most common logging daemons used by Linux distributions, and is the default syslog server in CentOS. Normally messages are logged to disk; however sending logs to a central location is often desirable from a security point of view. Collating logs from multiple systems also makes them easier to query.

This post is going to go over using Rsyslog to sent and receive messages across a network. For the example below I'm going to use two CentOS 7 virtual machines with the following IP addresses:

client.localdomain: 10.0.2.7
server.localdomain: 10.0.2.8

UDP delivery

UDP delivery is probably the simplest way to set up remote logging. On the server receiving the messages add the following configuration:

# Enable logging via UDP
$ModLoad imudp

# Listen of port 514
$UDPServerRun 514

# Specify the path remote messages should be written to
$template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log"

# Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from
# localhost
if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG
& end

Note: this configuration can either be added to the main configuration file (/etc/rsyslog.conf), or a file under /etc/rsyslog.d (e.g. /etc/rsyslog.d/remote.conf).

The client configuration just needs the following line:

*.info  @10.0.2.8:514

This tells Rsyslog on the client to forward all messages which are info level, or higher onto 10.0.2.8 via UDP.

Once the configuration has been updated on the client and the server, restart Rsyslog to re-read the configuration:

systemctl restart rsyslog.service

Now the configuration is in place, the logger command can be used to send a test message from the client:

[root@client ~]# logger -p local0.info -t testing-testing "hello world"

If everything is working correctly, this message should be logged on the remote system:

[root@server tls]# cat /var/log/remote/10.0.2.7/syslog.log
Feb 12 09:47:17 client testing-testing: hello world

TCP delivery

TCP can be used instead of UDP, this ensures messages are delivered in order, however it can also cause problems if the remote syslog server is not available for any reason. A post on the Rsyslog site goes over some of the considerations when picking TCP or UDP.

To enable logging over TCP, the server configuration needs to use the imtcp module:

# Enable logging via TCP
$ModLoad imtcp

# Listen of port 514
$InputTCPServerRun 514

# Specify the path remote messages should be written to
$template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log"

# Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from
# localhost
if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG
& end

The client configuration also needs to be updated to use @@ instead of @, so messages are sent using TCP:

*.info  @@10.0.2.8:514

Note: don't forget to restart Rsyslog on the server and client.

Encrypting messages

By default log messages are sent in the clear. This is fine if you trust your network, however it is possible to encrypt messages using rsyslog-gnutls. The first thing to do is install the rsyslog-gnutls package:

yum install rsyslog-gnutls

Once the package is installed, generate x509 public/private keys on the server:

openssl genrsa  -out /etc/pki/tls/private/rsyslog-key.pem 2048
openssl req -x509 -new -key /etc/pki/tls/private/rsyslog-key.pem \
  -out /etc/pki/tls/certs/rsyslog.pem -days 365

The server configuration then needs to be told about the new certificates, and the gtls driver needs to be enabled:

# Set x509 certificate paths
$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/rsyslog.pem
$DefaultNetstreamDriverCertFile /etc/pki/tls/certs/rsyslog.pem
$DefaultNetstreamDriverKeyFile /etc/pki/tls/private/rsyslog-key.pem

# Enable logging via TCP
$ModLoad imtcp

# Listen of port 514
$InputTCPServerRun 514

# Enable TLS encryption
$InputTCPServerStreamDriverMode 1
$InputTCPServerStreamDriverAuthMode anon

# Specify the path remote messages should be written to
$template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log"

# Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from
# localhost
if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG
& end

Once the server is configured, copy across the server certificate to the client and update the client config:

$DefaultNetstreamDriver gtls
$DefaultNetstreamDriverCAFile /etc/pki/tls/certs/rsyslog.pem
$ActionSendStreamDriverMode 1
$ActionSendstreamDriverAuthMode anon

*.info  @@10.0.2.8:514

Once Rsyslog is restarted on the server and client, all syslog traffic should now be encrypted.

Floating Octothorpe

Remote logging with Rsyslog

UDP delivery

TCP delivery

Encrypting messages

Further reading