Log rotation is a fundamental Linux operation that’s often overlooked until it causes disk space issues or makes debugging impossible.
Let’s see logrotate in action. Imagine you have a web server, say Nginx, generating logs. We want to compress old logs, keep a few recent ones, and make sure the current log file is empty and ready for new entries.
Here’s a basic logrotate configuration for Nginx, typically found in /etc/logrotate.d/nginx:
/var/log/nginx/*.log {
daily
missingok
rotate 14
compress
delaycompress
notifempty
create 0640 www-data adm
sharedscripts
postrotate
if [ -f /var/run/nginx.pid ]; then
kill -USR1 `cat /var/run/nginx.pid`
fi
endscript
}
This block tells logrotate how to handle any file ending in .log within the /var/log/nginx/ directory.
daily: This directive meanslogrotatewill attempt to run this configuration once a day.logrotateitself is usually triggered by a cron job, often daily.missingok: If the log file doesn’t exist,logrotatewon’t complain and will just move on. This is useful if a service might not be running or if log files are sometimes removed manually.rotate 14: This is the core of rotation. It meanslogrotatewill keep 14 old log files. When a new rotation happens and the 15th file would be created, the oldest one (file 1) is deleted.compress: This tellslogrotateto compress the rotated log files. By default, it usesgzip. The rotated files will have a.gzextension.delaycompress: This is a subtle but important one. It meanslogrotatewill compress the previous log file, not the one that was just rotated. So, if you haveaccess.log,access.log.1,access.log.2, andcompressis active,access.log.1will be compressed toaccess.log.1.gzonly on the next rotation cycle. This is useful if some process might still be writing to the log file that was just rotated but hasn’t been reopened yet.notifempty: If a log file is empty,logrotatewill not rotate it. This prevents creating empty rotated log files when a service is idle.create 0640 www-data adm: After rotating the current log file (e.g.,access.logbecomesaccess.log.1), this directive tellslogrotateto create a new, emptyaccess.logfile with specific permissions (0640), owner (www-data), and group (adm). This ensures the service can immediately start writing to the new log file.sharedscripts: This is crucial for multiple logs matching the pattern. It ensures that thepostrotatescript runs only once after all logs matching the pattern have been rotated, rather than once for each log file.postrotate/endscript: This block contains commands that are executed after the log file has been rotated. For Nginx, theUSR1signal is sent to the master process. This signal tells Nginx to reopen its log files, meaning it will start writing to the newly created empty log file. Without this, Nginx would continue writing to the old, rotated file descriptor until it’s restarted.
The system logrotate operates within is driven by cron. Typically, a cron job in /etc/cron.daily/logrotate executes the logrotate command with the -f (force) flag if a configuration file specifies a run frequency that has passed, or just logrotate which checks its configuration files. The logrotate daemon then scans /etc/logrotate.conf and all files in /etc/logrotate.d/ to determine which logs need rotation based on the directives within each configuration file.
A common pitfall is forgetting the postrotate script. If you rotate a log file but the application writing to it doesn’t know, it will continue writing to the old file descriptor. This means the new, empty file created by logrotate will remain empty, and the old file will continue to grow indefinitely, defeating the purpose of rotation.
The one thing that often trips people up is how delaycompress interacts with compress. If you have a log file that rotates daily, compress will compress access.log.1 into access.log.1.gz immediately after rotation. However, delaycompress means it won’t compress access.log.1 until the next day, when it becomes access.log.2. This ensures that the most recently rotated file (access.log.1) is uncompressed and easily accessible for immediate inspection if needed, while older logs are compressed.
After fixing log rotation issues, you might encounter problems with filesystems becoming read-only due to excessive inode usage if you’ve set rotate to a very high number or are dealing with an extremely verbose application.