Centralizing Logs from Multiple Servers into One

Imagine the following scenario: you have N frontend servers, and each generates its own logs (access, error, PHP, slow) and stores them locally in /var/log/nginx. The day an incident occurs and you need to investigate, it may be that only one node is causing the problem, but you would have to check each server individually.

To mitigate this, the ideal solution is to aggregate logs from all servers into a single server. This way, you can see logs from all servers in one place and determine where the issue originates.

I analyzed several alternatives for this task. The ones that caught my attention were Graylog2 and Logstash, but both have complex architectures and are aimed at advanced log processing. They’re worth checking out, but for now, I skipped them. Another tool often recommended is Splunk, but I didn’t get to try it because it’s paid.

I went back to the basics: rsyslog, which comes by default as syslog on many Linux distributions. I already had it installed, so I decided to use it. The goal was to configure one server to receive logs and have the frontend nodes send their logs.

The first step is to configure rsyslog to listen. Edit /etc/rsyslog.conf and enable the modules for TCP listening:

#################
#### MODULES ####
#################
$ModLoad imuxsock    # provides support for local system logging
$ModLoad imklog      # provides kernel logging support
$ModLoad imudp       # provides UDP syslog reception
$UDPServerRun 514
$ModLoad imtcp       # provides TCP syslog reception
$InputTCPServerRun 514

These lines may be commented out; uncomment them.

The second step is to create a .conf file in /etc/rsyslog.d/. All .conf files in this folder will be included in the rsyslog configuration. Let’s call the file servers.conf. It should define what to do with each incoming log stream:

*.*;auth,authpriv.none,local0.none -/var/log/syslog
local0.* -/var/log/servers/nginx.access.log
*.*;auth,authpriv.none,local1.none -/var/log/syslog
local1.* -/var/log/servers/nginx.error.log
*.*;auth,authpriv.none,local2.none -/var/log/syslog
local2.* -/var/log/servers/php-error.log
*.*;auth,authpriv.none,local3.none -/var/log/syslog
local3.* -/var/log/servers/www-slow.log

Rsyslog manages each stream with a local identifier [0-7], so here we use the first four. Aggregated logs will be stored in /var/log/servers. After restarting rsyslog, the server should be ready to receive logs.

The client setup is simpler. On each node, create a .conf file in /etc/rsyslog.d/. Let’s call it client.conf. It should include the following:

$ModLoad imfile
$InputFileName /var/log/nginx/[your_access_log]
$InputFileTag access:
$InputFileFacility local0
$InputFileSeverity info
$InputRunFileMonitor

$InputFileName /var/log/nginx/[your_error_log]
$InputFileTag nginx_error:
$InputFileFacility local1
$InputFileSeverity info
$InputRunFileMonitor

$InputFileName /var/log/php-fpm/[your_php_error]
$InputFileTag php_error:
$InputFileFacility local2
$InputFileSeverity info
$InputRunFileMonitor

$InputFileName /var/log/php-fpm/www-slow.log
$InputFileTag php_slow:
$InputFileFacility local3
$InputFileSeverity info
$InputRunFileMonitor
$InputFilePollInterval 1

local0.* @(z7)[SERVER_IP]:514
local1.* @(z7)[SERVER_IP]:514
local2.* @(z7)[SERVER_IP]:514
local3.* @(z7)[SERVER_IP]:514

Note: Replace placeholders with your actual file paths and server IP.

After restarting the server, if you run tail -f on any log file, you should see logs being forwarded from the client nodes. Make sure all nodes are sending activity to the log server.

Common problems are usually related to network addresses (machines can’t reach each other) or firewall issues. This configuration is for Nginx and PHP-FPM, but the logic can be applied to forward any log file.

Thanks to the DevOps community for the tips—they suggested these tools.