Analyzing System Logs with journalctl and /var/log

The Big Picture: How Linux Logging Works

Before we look at commands, it helps to understand where logs come from. A Linux system is constantly generating messages from the kernel (about hardware and system state) and from user-space applications and services (like SSH, cron, or a web server). These messages need to be collected and stored.

1. The Classic Approach: Traditional Logs in /var/log

For decades, the standard for logging on Linux and Unix systems has been to write plain-text files into the /var/log directory. Each file or subdirectory typically serves a specific purpose.

To get a great overview of this system, including how log file names can differ between Linux families (like Red Hat vs. Debian), let's watch the first part of the following video.

Key Log Files in /var/log

As you saw in the video, different Linux distributions might name their log files slightly differently. Here is a table summarizing the most common ones. This is also covered well in the article "Log Management with journalctl and /var/log" under the section "Classic Logs in /var/log/".

Tools for Reading Text Logs

You already know most of these! The power comes from combining them.

• less /var/log/syslog: View a large log file with pagination.
• tail /var/log/auth.log: See the last 10 lines of the authentication log.
• tail -f /var/log/auth.log: Follow the log in real-time. This is extremely useful for live troubleshooting.
• grep "Failed password" /var/log/auth.log: Search for specific patterns within a log file.
• grep "CRON" /var/log/syslog | tail -n 20: Find all lines related to cron in the system log and show only the last 20. This is how you can check if your scheduled jobs ran!

2. The Modern Approach: systemd and journalctl

Most modern Linux distributions use systemd as their init system. One of systemd's core components is the journal, a centralized and structured logging service. Instead of writing to many different text files, all logs—from the kernel, services, and applications—are collected by the journald daemon and stored in a binary format.

You interact with the journal using the journalctl command.

Why journalctl?

• Centralized: All logs are in one place.
• Structured Data: Each log entry is tagged with metadata (e.g., the service it came from, its priority, the process ID).
• Powerful Filtering: You can easily filter logs by service, time range, priority, and more, which is much harder with plain text files.

This next video gives a focused tutorial on the most important features of journalctl.

Core journalctl Commands

The video you just watched demonstrated the essentials. Here's a quick-reference summary. For an even more detailed command list, you can refer to the "Log Management with journalctl" article.

Common journalctl Commands:

• journalctl: View all logs (from oldest to newest).
• journalctl -f: Follow logs in real-time.
• journalctl -n 50: Show the last 50 log entries.
• journalctl -u sshd: Show all logs specifically from the sshd service.
• journalctl -u nginx --since "10 minutes ago": Show Nginx logs from the last 10 minutes.
• journalctl -b: Show all logs from the current system boot. Use -b -1 for the previous boot.
• journalctl -p err: Show all logs with a priority of "error" or higher (err, crit, alert, emerg).
• journalctl -k: Show only kernel messages (similar to the dmesg command).

Practical Scenarios: When to Use Which

So, when do you use journalctl versus peeking into /var/log?

• "My Nginx web server won't start."
  • Best tool: journalctl -u nginx.service. This isolates logs to just the service you're debugging. Adding -xe (journalctl -xe -u nginx.service) can give even more detailed error information.
• "Did someone try to SSH into my server last night?"
  • Both work!
    • grep "sshd" /var/log/auth.log is a classic.
    • journalctl -u sshd --since yesterday is the modern, potentially more precise way.
• "My custom application is failing, and it writes its logs to /var/log/my-app/error.log."
  • Best tool: tail -f /var/log/my-app/error.log. Since this application doesn't log to the journal, you must use the traditional text file.
• "The system just rebooted unexpectedly. Why?"
  • Best tool: journalctl -b -1 -p 3. This shows you logs from the previous boot (-b -1) with a priority of critical or higher (-p 3 or -p crit). This can quickly point you to the failure that occurred right before the crash.

Conclusion

You have now added a critical diagnostic skill to your DevOps toolkit. You can confidently investigate system behavior, security events, and application errors.

Key Takeaways:

• Linux has two main logging systems: traditional text files in /var/log and the systemd journal.
• Use commands like tail, less, and grep to read and search through files in /var/log. tail -f is essential for live troubleshooting.
• Key traditional logs include /var/log/syslog (general), /var/log/auth.log (security), and /var/log/apt/history.log (packages).
• journalctl is the powerful command-line tool for querying the systemd journal.
• The real power of journalctl lies in its filtering capabilities: by unit (-u), time (--since), priority (-p), and boot (-b).
• For debugging systemd services like Nginx or SSH, journalctl -u <service> is your go-to command.

This lesson concludes our module on Linux System Administration Essentials. You've built a strong foundation, from managing services and users to scheduling tasks and now, analyzing logs.

Our next module, "Networking Fundamentals for DevOps," begins with a lesson on the OSI model. Understanding how data flows across a network is the first step toward configuring and troubleshooting the complex, interconnected systems you'll be managing as a DevOps engineer.