Remote logging with Rsyslog
Rsyslog is one of the most common logging daemons used by Linux distributions, and is the default syslog server in CentOS. Normally messages are logged to disk; however sending logs to a central location is often desirable from a security point of view. Collating logs from multiple systems also makes them easier to query.
This post is going to go over using Rsyslog to sent and receive messages across a network. For the example below I'm going to use two CentOS 7 virtual machines with the following IP addresses:
- client.localdomain: 10.0.2.7
- server.localdomain: 10.0.2.8
UDP delivery is probably the simplest way to set up remote logging. On the server receiving the messages add the following configuration:
# Enable logging via UDP $ModLoad imudp # Listen of port 514 $UDPServerRun 514 # Specify the path remote messages should be written to $template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log" # Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from # localhost if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG & end
Note: this configuration can either be added to the main configuration file
/etc/rsyslog.conf), or a file under
The client configuration just needs the following line:
This tells Rsyslog on the client to forward all messages which are info level,
or higher onto
10.0.2.8 via UDP.
Once the configuration has been updated on the client and the server, restart Rsyslog to re-read the configuration:
systemctl restart rsyslog.service
Now the configuration is in place, the logger command can be used to send a test message from the client:
[[email protected] ~]# logger -p local0.info -t testing-testing "hello world"
If everything is working correctly, this message should be logged on the remote system:
[[email protected] tls]# cat /var/log/remote/10.0.2.7/syslog.log Feb 12 09:47:17 client testing-testing: hello world
TCP can be used instead of UDP, this ensures messages are delivered in order, however it can also cause problems if the remote syslog server is not available for any reason. A post on the Rsyslog site goes over some of the considerations when picking TCP or UDP.
To enable logging over TCP, the server configuration needs to use the
# Enable logging via TCP $ModLoad imtcp # Listen of port 514 $InputTCPServerRun 514 # Specify the path remote messages should be written to $template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log" # Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from # localhost if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG & end
The client configuration also needs to be updated to use
@@ instead of
so messages are sent using TCP:
Note: don't forget to restart Rsyslog on the server and client.
By default log messages are sent in the clear. This is fine if you trust your
network, however it is possible to encrypt messages using
first thing to do is install the
yum install rsyslog-gnutls
Once the package is installed, generate x509 public/private keys on the server:
openssl genrsa -out /etc/pki/tls/private/rsyslog-key.pem 2048 openssl req -x509 -new -key /etc/pki/tls/private/rsyslog-key.pem \ -out /etc/pki/tls/certs/rsyslog.pem -days 365
The server configuration then needs to be told about the new certificates, and the gtls driver needs to be enabled:
# Set x509 certificate paths $DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/pki/tls/certs/rsyslog.pem $DefaultNetstreamDriverCertFile /etc/pki/tls/certs/rsyslog.pem $DefaultNetstreamDriverKeyFile /etc/pki/tls/private/rsyslog-key.pem # Enable logging via TCP $ModLoad imtcp # Listen of port 514 $InputTCPServerRun 514 # Enable TLS encryption $InputTCPServerStreamDriverMode 1 $InputTCPServerStreamDriverAuthMode anon # Specify the path remote messages should be written to $template REMOTE_SYSTEM_LOG,"/var/log/remote/%fromhost-ip%/syslog.log" # Log messages to the `REMOTE_SYSTEM_LOG` path if they are not from # localhost if ($fromhost-ip != '127.0.0.1') then ?REMOTE_SYSTEM_LOG & end
Once the server is configured, copy across the server certificate to the client and update the client config:
$DefaultNetstreamDriver gtls $DefaultNetstreamDriverCAFile /etc/pki/tls/certs/rsyslog.pem $ActionSendStreamDriverMode 1 $ActionSendstreamDriverAuthMode anon *.info @@10.0.2.8:514
Once Rsyslog is restarted on the server and client, all syslog traffic should now be encrypted.
If you're after more information on configuring Rsyslog, the official documentation is a good place to start. If you're interested the syslog protocol, RFC 3164 is worth a read, even if it's just for the creative intro:
Since the beginning, life has relied upon the transmission of messages. For the self-aware organic unit, these messages can relay many different things. The messages may signal danger, the presence of food or the other necessities of life, and many other things...