Automating Log Rotation with Bash

1. What is Log Rotation?

Log rotation is the process of managing log files to prevent them from growing indefinitely. A typical rotation process involves:

1. Archiving the current active log file (e.g., renaming app.log to app.log.1).
2. Creating a new, empty log file for the application to write to (app.log).
3. Optionally compressing the archived log files to save space (e.g., app.log.1.gz).
4. Deleting the oldest archived log files after a certain number of rotations to manage long-term disk usage.

This process ensures that log files remain small and manageable, making analysis easier and preventing disk space exhaustion.

The image below provides a great visual summary of this cycle.

2. The Manual Approach: A Custom Bash Script

To understand the mechanics of log rotation, let's start by writing a simple Bash script. While not as robust as standard tools, this approach directly addresses the "using a Bash script" part of our goal and reveals the underlying logic.

One common trigger for rotation is file size. The script below will check if a log file has exceeded a specific size and, if so, rename it with a timestamp.

Let's break down the core logic of that script:

#!/bin/bash

LOGFILE="/var/log/my-app.log"
MAX_SIZE_MB=1 # Set max size in MB

# Check if the log file exists and is a file
if [ -f "$LOGFILE" ]; then
    # Get the size of the log file in megabytes
    log_size_mb=$(du -m "$LOGFILE" | awk '{print $1}')

    # If the log file is larger than MAX_SIZE_MB, rotate it
    if [ "$log_size_mb" -gt "$MAX_SIZE_MB" ]; then
        echo "Log file is larger than ${MAX_SIZE_MB}MB. Rotating."
        # Move the current log to a new file with a timestamp
        mv "$LOGFILE" "$LOGFILE.$(date +"%Y%m%d%H%M%S")"
        # Create a new empty log file for the application to use
        touch "$LOGFILE"
    else
        echo "Log file size is within limits."
    fi
else
    echo "Log file does not exist. Creating it."
    touch "$LOGFILE"
fi

How it works:

• du -m "$LOGFILE": The du (disk usage) command with the -m flag reports the file size in megabytes.
• awk '{print $1}': This command extracts the first column (the size number) from the output of du.
• if [ "$log_size_mb" -gt "$MAX_SIZE_MB" ]: A standard Bash conditional that checks if the log size is greater than our defined maximum.
• mv "$LOGFILE" "$LOGFILE.$(date ...)": If the condition is met, the mv command renames the log file, appending a timestamp just like we did for our backups.
• touch "$LOGFILE": This creates a new, empty log file with the original name so the application can continue logging without interruption.

This script demonstrates the fundamental concept, but it has limitations: it doesn't compress old logs, doesn't delete the oldest archives, and requires its own cron job to run. For production systems, there's a better way.

3. The Professional Approach: Using logrotate

On virtually all Linux systems, the standard tool for managing logs is logrotate. It is a highly flexible and powerful utility that runs automatically, managing log files based on a set of declarative rules. As a DevOps engineer, logrotate will be your primary tool for this task.

Let's walk through a project to configure logrotate for a custom application.

The logrotate Configuration

Now that you have a test log file, the next step is to tell logrotate how to handle it. This is done by creating a simple configuration file. The main configuration for logrotate is /etc/logrotate.conf, but for individual applications, we add files to the /etc/logrotate.d/ directory. This keeps our system tidy and makes it easy to manage configurations for different services.

Let's examine how to write one of these configuration files.

A typical logrotate configuration file for a web server log might look like this:

/var/log/nginx/*.log {
    daily
    missingok
    rotate 14
    compress
    delaycompress
    notifempty
    create 0640 www-data adm
    sharedscripts
    postrotate
        if [ -f /var/run/nginx.pid ]; then
            kill -USR1 `cat /var/run/nginx.pid`
        fi
    endscript
}

Let's review the key directives:

• /var/log/nginx/*.log: This defines which log files the rules apply to. Wildcards are supported.
• daily: Rotate logs once a day. Other options include weekly, monthly, or size 100M.
• rotate 14: Keep 14 of the most recent archived logs. Older ones will be deleted.
• compress: Compress the rotated log files using gzip.
• notifempty: Do not rotate the log file if it is empty.
• create 0640 www-data adm: After rotating, create a new log file with the specified permissions (0640), owner (www-data), and group (adm).
• postrotate/endscript: This block contains a script to be run after the log file has been rotated. This is crucial for many applications, which need to be signaled to close their old log file handle and open the new one. For Nginx, sending the USR1 signal tells it to re-open its log files.

/opt/app/app.log {
    size 50M
    rotate 5
    compress
    missingok
    notifempty
}

Testing and Automating logrotate

With your configuration file in place, the final steps are to test it and ensure it runs automatically.

Key Commands for Testing:

• Dry Run: sudo logrotate -d /etc/logrotate.d/your_config_file
  • The -d (debug) flag makes logrotate perform a "dry run." It will show you exactly what it would do without actually modifying any files. This is perfect for verifying your syntax and logic.
• Forced Run: sudo logrotate -f /etc/logrotate.d/your_config_file
  • The -f (force) flag forces logrotate to run, even if the conditions (like daily or size) haven't been met. This is useful for immediately testing that the entire process works as expected.

Automation:
As the article notes, you almost never need to set up a cron job for logrotate yourself. Most Linux distributions, including Ubuntu and CentOS, ship with a pre-configured daily cron job (often located at /etc/cron.daily/logrotate) that runs logrotate on all configuration files in /etc/logrotate.d/.

By simply placing your configuration file in that directory, you have already automated the process.

Conclusion

Congratulations! You've mastered another fundamental automation task that is essential for maintaining healthy, stable systems. You now understand not only the "how" but also the "why" behind log rotation.

Key Takeaways:

• Log rotation is essential to prevent log files from consuming all disk space and to keep them manageable.
• A basic Bash script can perform simple rotation (e.g., based on size), which helps in understanding the core mechanics.
• logrotate is the industry-standard tool for this job. It is powerful, flexible, and uses a declarative configuration.
• To manage logs for a new application, you simply add a configuration file to /etc/logrotate.d/.
• Automation is built-in. A system cron job typically runs logrotate daily, so no manual scheduling is usually required.

Next Up

In this module, we've built several practical scripts for monitoring, backups, and now, log management. These projects have given you a taste of the power of Bash. To truly master automation, however, you need a deeper understanding of the scripting language itself.

In our next lesson, we will begin the next module, "Scripting and Automation with Bash," by diving into the fundamental building blocks of the language. We will formally cover control structures like if, for, and while loops, giving you the structured programming tools needed to write more complex and intelligent scripts.