Journald: Persistent Logging and Real-Time Monitoring

1. Enabling Persistent Logging

The systemd-journald service can store logs in two ways:

• Volatile: Logs are kept in memory (specifically in the /run/log/journal/ directory). They are fast but are lost when the system reboots.
• Persistent: Logs are written to the disk (in the /var/log/journal/ directory). They survive reboots, creating a historical record of your system's activity.

Fedora typically enables persistent logging by default on new installations, but it's a critical concept to understand and verify. The system's behavior is controlled by a simple rule: if the /var/log/journal directory exists, journald will use it for persistent storage.

For a clear, practical explanation of how this works and how to enable it if it's not already active, please read the following guide.

Do-It-Yourself: Verify and Test Persistence

Let's apply what you just read. We will check your system's current configuration and ensure it's set for persistence.

1. Check for the Journal Directory:
   First, check if the persistent log directory exists on your system.

   ls /var/log/journal
   

   If this command shows a directory with a long alphanumeric name (your machine's ID), then persistent logging is likely already active. If it reports No such file or directory, you would need to create it with sudo mkdir -p /var/log/journal.

2. List System Boots:
   The definitive test is to ask journalctl which boot sessions it has logs for.

   journalctl --list-boots
   

   The output shows one line per boot session. A 0 in the first column refers to the current boot. A -1 refers to the previous boot, -2 the one before that, and so on. If you see more than just the current boot (0), your logs are persistent!

3. The Ultimate Test (Optional but Recommended):
   To see this in action, reboot your machine.

   sudo reboot
   

   After you log back in, open a terminal and run journalctl --list-boots again. You should now see your previous session listed with a -1 offset. You can even view its logs with journalctl -b -1. This confirms your system is keeping a historical record.

2. Managing Log Size

With great power comes great responsibility. Storing logs forever would eventually fill up your hard drive. journald has built-in mechanisms to control how much disk space the logs are allowed to consume.

You can configure this in the file /etc/systemd/journald.conf or manually clean up old logs using journalctl itself. The following video and text provide a great overview.

To summarize the key points:

• Configuration File: In /etc/systemd/journald.conf, you can uncomment and set options like SystemMaxUse= (e.g., SystemMaxUse=1G) to limit the total space used by logs, or SystemKeepFree= to ensure a certain amount of disk space is always available.
• Manual Cleanup (Vacuuming): You can clean up archived journal files on demand. This is useful for immediate space recovery. The main commands are:
  • --vacuum-size=SIZE: Deletes old logs until the total size is under SIZE (e.g., 500M).
  • --vacuum-time=TIME: Deletes logs older than TIME (e.g., 2d for two days, 1w for one week).

Do-It-Yourself: Check and Clean

Let's practice managing your journal's size.

1. Check Current Disk Usage:
   Find out how much space your journal logs are currently using.

   journalctl --disk-usage
   

2. Perform a Cleanup:
   Let's run a cleanup operation. We'll use the --vacuum-size option to shrink the journal archives to a specific size. Let's aim for 500MB.

   sudo journalctl --vacuum-size=500M
   

   The command will report how much space was freed.

3. Verify the Result:
   Check the disk usage again to see the effect of the vacuuming operation.

   journalctl --disk-usage
   

   Note: The total size might still be slightly above your target because the vacuum command primarily affects archived journal files, not the active one currently being written to.

3. Following Logs in Real-Time

In the last lesson, we briefly mentioned the -f (or --follow) flag. Now, let's put it into a practical workflow. Following a log file is one of the most common activities when debugging a service. You get to see new log messages appear on your screen the instant they are generated.

This is best demonstrated with a real service. We'll continue using the httpd web server.

Do-It-Yourself: Live Debugging Simulation

For this exercise, I recommend opening two terminal windows and placing them side-by-side.

1. In Terminal 1 (The "Observer"):
   Start following the logs for the httpd service. This terminal will now be your live log viewer.

   journalctl -f -u httpd.service
   

   Your cursor will wait at the end of the output.

2. In Terminal 2 (The "Actor"):
   Use this terminal to issue commands that affect the service. After each command, look at Terminal 1 to see the log messages that appear.

   • Stop the service:

     sudo systemctl stop httpd
     

     You should see messages in Terminal 1 indicating that httpd.service: Succeeded. or that the service was stopped.

   • Start the service again:

     sudo systemctl start httpd
     

     Now you'll see startup messages appear in Terminal 1.

   • Restart the service:

     sudo systemctl restart httpd
     

     You'll see a quick sequence of stopping and starting messages.

3. When you're finished, go back to Terminal 1 and press Ctrl+C to stop following the log.

This two-terminal workflow is fundamental for system administration and development. You'll use it constantly when deploying your Django projects to see if they start correctly or to diagnose errors when a user accesses a web page.

Conclusion

Congratulations on completing this module on process and service management! You have now mastered the essential tools for controlling and observing the software running on your Fedora system.

Key Takeaways:

• Logs can be volatile (in memory, lost on reboot) or persistent (on disk, survive reboots).
• Persistent logging is enabled by ensuring the /var/log/journal directory exists and restarting the systemd-journald service.
• You must manage the size of persistent logs, either through the /etc/systemd/journald.conf file or with manual journalctl --vacuum-* commands.
• The journalctl -f command is your tool for real-time monitoring, a crucial part of any debugging workflow.

Preview of the Next Module:

So far, we've focused on what's happening inside your machine. In our next module, "Network Configuration and Security," we'll turn our attention to how your system communicates with the outside world. You will learn how to check your IP address, test network connectivity, and configure the firewall—all critical skills for making your future web applications accessible and secure on the internet.