The most surprising thing about rsyslog is that it’s not just a dumb log forwarder; it’s a sophisticated, stateful message broker that can queue, filter, transform, and even rate-limit log messages before they hit disk or network.
Let’s see it in action. Imagine you have a fleet of application servers (let’s call one appserver-01) and you want all their logs to go to a central log server (logserver.example.com).
On appserver-01, you’d have a basic rsyslog.conf (often in /etc/rsyslog.conf or /etc/rsyslog.d/). To send logs to logserver.example.com on port 514 using UDP (a common, though less reliable, choice for speed), you’d add a line like this:
*.* @logserver.example.com:514
The *.* means "all facilities, all priorities" (i.e., every log message). The @ symbol signifies UDP. If you wanted TCP (more reliable, but slower and with more overhead), you’d use @@.
Now, on logserver.example.com, you need to configure rsyslog to receive these logs. You’d typically create a new file in /etc/rsyslog.d/, say 50-remote.conf, with the following:
module(load="imudp")
input(type="imudp" port="514")
module(load="imtcp")
input(type="imtcp" port="514") # For TCP, if you used @@ from clients
This loads the UDP input module and tells it to listen on port 514. It also loads the TCP module, just in case.
But where do these received logs go? By default, they might just get mixed in with local logs, which is messy. We want to separate them. So, in 50-remote.conf on logserver.example.com, we’d add rules to direct logs from specific hosts or based on their content to separate files.
Let’s say we want all logs from appserver-01 to go into /var/log/remote/appserver-01.log. We’d add these lines after the input modules:
$template RemoteHostLog,"/var/log/remote/%HOSTNAME%.log"
*.* ?RemoteHostLog
The $template RemoteHostLog defines a template named RemoteHostLog that specifies the file path. %HOSTNAME% is a rsyslog variable that will be replaced by the hostname of the machine sending the log. The *.* ?RemoteHostLog rule says "take all messages (*.*) and direct them to the file specified by the RemoteHostLog template."
After adding these configurations, you’d restart rsyslog on both servers:
On appserver-01:
sudo systemctl restart rsyslog
On logserver.example.com:
sudo systemctl restart rsyslog
Now, when appserver-01 generates a log message (e.g., an auth.log entry), it will be sent to logserver.example.com and appear in /var/log/remote/appserver-01.log.
The true power comes when you start adding filters and more complex routing. For instance, on logserver.example.com, you might want to send only auth facility logs from a specific server to a separate, more secure location, or perhaps drop debug messages entirely to save space.
# In /etc/rsyslog.d/50-remote.conf on logserver.example.com
# ... (input modules and general remote logging as before) ...
# Specific rule for critical logs from appserver-01
if $fromhost-ip == '192.168.1.101' and $syslogfacility-text == 'auth' then /var/log/auth_special.log
# Drop all debug messages from anywhere
if $syslogseverity-text == 'debug' then stop
Here, the if ... then ... structure allows for conditional logic. $fromhost-ip checks the IP address of the sender (you could also use $fromhost for the hostname). $syslogfacility-text checks the facility name. stop tells rsyslog to cease processing this message further down the rule chain.
rsyslog also has robust support for queuing. If your logserver.example.com becomes temporarily unavailable, you can configure appserver-01 to queue messages locally and send them once the connection is restored, preventing log loss. This is done via actionqueue properties:
# On appserver-01, in /etc/rsyslog.d/50-remote.conf
*.* action(type="omfwd" target="logserver.example.com" port="514" protocol="udp"
queue.type="LinkedList" # Or "Disk" for persistent queue
queue.filename="fwd_queue" # For disk-assisted queuing
queue.size="10000" # Max number of messages in queue
queue.saveonshutdown="on" # Save queue on shutdown
queue.maxdiskspace="1g" # Max disk space for queue
action.resumeinterval="30" # Try to reconnect every 30 seconds
action.resumeonstartup="on") # Resume sending on startup
This configuration uses omfwd (output module for forwarding) and defines a linked list queue. If you wanted disk-based queuing for even greater resilience, you’d set queue.type="Disk". The action.resumeinterval and action.resumeonstartup are crucial for handling temporary network glitches.
What most people don’t realize is how rsyslog handles message processing order. Rules are evaluated sequentially from top to bottom within a configuration file, and within the rsyslog.conf and files in /etc/rsyslog.d/. A stop action on a rule will prevent subsequent rules from being evaluated for that message. This means the order of your rules is paramount for complex filtering and routing. If you have a general rule for forwarding all logs, but a specific rule later to drop certain messages, the general rule might send them before the drop rule gets a chance to act unless the specific rule uses stop.
The next step is often implementing more advanced filtering based on message content, using regular expressions, or setting up high-availability for your central log server.