Navigating System Logs with `journalctl` and `/var/log`

The Two Worlds of Linux Logging

Modern Linux systems, including Arch, typically use two logging systems that work in tandem. It's crucial to understand both.

1. rsyslog (and similar): The traditional, file-based system. It writes logs as plain text files into the /var/log directory. This makes them easy to read and process with standard command-line tools.
2. journald: The newer system introduced by systemd. It captures logs from all over the system (including kernel, services, etc.) and stores them in a structured, indexed binary format. You interact with it using the journalctl command.

Let's begin by reading a short article that explains how these two systems coexist.

Now that you have the theoretical overview, let's dive into the practical side, starting with the modern journalctl.

Part 1: The systemd Journal (journalctl)

Since we just covered systemd and systemctl, journalctl is the natural next step. It's an incredibly powerful tool for querying the logs that systemd and its services produce.

Filtering: The Real Power of journalctl

The true strength of journalctl lies in its ability to quickly filter the vast amount of log data. Because the logs are structured, you can slice and dice them in powerful ways.

Filtering by Service (Unit)

This is the most direct link to our previous lesson. If you want to see the logs for a specific service, you use the -u (for "unit") flag.

Filtering by Time

Often, you only care about logs from a specific time frame, like "since yesterday" or "in the last hour."

Other Useful Filters

There are many other ways to filter. Here are a few essential ones to know:

• journalctl -p <priority>: Filter by message priority. Priorities range from emerg (0) to debug (7). Using -p err will show all messages with priority "error" and higher (crit, alert, emerg). This is invaluable for finding problems quickly.
• journalctl -b: Show messages from the current boot only.
• journalctl -b -1: Show messages from the previous boot.
• journalctl -k: Show only kernel messages (similar to the dmesg command).

Part 2: Traditional Logs in /var/log

While journalctl is the modern approach, you absolutely must be familiar with the classic text-based logs in the /var/log directory. Many applications log here by default, and these files are your target for analysis on systems that don't use systemd.

Key Files to Know

The /var/log directory contains many files. Here are some of the most important for security and troubleshooting:

Tools for Reading and Searching Text Logs

Since these are plain text files, we can use standard Linux utilities.

• less <file>: The best tool for viewing large log files. It lets you scroll up and down and search for text.
• tail -f <file>: This is the classic way to "follow" a log file in real-time. New entries are printed to your screen as they are written to the file.
• grep <pattern> <file>: Your primary tool for searching text files. You can search for keywords like "error", "failed", or a specific IP address.

Let's see a practical demonstration.

A Note on Log Rotation

You might notice files in /var/log like auth.log.1, auth.log.2.gz, etc. Log files can't grow forever, or they would fill up the disk. A process called log rotation automatically renames and compresses old logs to save space.

• auth.log: The current, active log file.
• auth.log.1: The most recent "retired" log file.
• auth.log.2.gz: An older log file that has been compressed with gzip. To read it, you'd need to use a tool like zless or zgrep.

Hands-on Practice

It's time to put this into practice on your own system. Open your terminal and let's get to work.

1. Investigate a Service with journalctl
   In the last lesson, you may have installed nginx. Let's assume it's installed.

   • Check its status: systemctl status nginx
   • Now, view its logs: journalctl -u nginx
   • Restart the service (sudo systemctl restart nginx) and immediately run journalctl -u nginx again. You should see the new log entries for the stop and start events.

2. Live Monitoring with journalctl
   Let's monitor SSH activity. Open two terminals.

   • Terminal 1: journalctl -fu sshd
   • Terminal 2: Try to SSH into your own machine: ssh your_username@localhost. Enter your password.
   • Watch the logs appear in Terminal 1. Now, try again but enter the wrong password. Observe the "Failed password" message. This is exactly what you'd look for when hunting for brute-force attacks.

3. Live Monitoring with tail and grep
   Let's do the same thing, but the classic way.

   • Terminal 1: tail -f /var/log/auth.log (You may need sudo to read this file).
   • Terminal 2: Use a sudo command, for example: sudo whoami.
   • Observe the new entry in Terminal 1 detailing your use of sudo.

4. Searching for Evidence
   Use grep to search for all failed login attempts on your system.

   
   
   
   # The -i flag makes the search case-insensitive
   grep -i "failed password" /var/log/auth.log
   

   This command is fundamental for security audits.

Conclusion

You now have the skills to query the two most important logging systems in Linux. This ability to look "under the hood" is what separates a casual user from a system administrator or security professional.

Key Takeaways:

• Two Systems: Modern Linux uses both the systemd journal (journalctl) and traditional text logs (/var/log).
• journalctl: Best for systemd services. Its power is in structured filtering by unit (-u), time (--since), priority (-p), and boot (-b).
• /var/log: Home of traditional text logs. Essential tools are less for viewing, tail -f for real-time monitoring, and grep for searching.
• Security Context: Logs are your primary source for detecting unauthorized access, troubleshooting service failures, and understanding system behavior after a potential compromise.

Next Lesson Preview:
After finding an issue in the logs—perhaps a service failing due to a misconfiguration—the next logical step is to fix it. In our next lesson, we will cover how to edit system configuration files using a terminal text editor like vim or nano, giving you the power to act on the information you've discovered.