System Log Analysis for Error Diagnosis

1. The Two Worlds of Linux Logging

To effectively find logs, you first need to understand that modern Linux systems typically use a hybrid approach to logging, combining a traditional method with a modern one.

• Traditional Syslog: This is the classic logging system. Services send their log messages to a daemon (like rsyslog), which then writes them as plain text files into the /var/log/ directory. Different types of messages are often sorted into different files.
• Systemd Journal: This is the modern, integrated logging system. The systemd-journald daemon collects logs from the kernel, system services, and applications into a centralized, binary-formatted journal. It offers powerful filtering and is accessed via the journalctl command.

Many systems run both, with journald capturing everything and forwarding it to rsyslog to be written to the traditional text files in /var/log. This gives you the best of both worlds: structured, queryable logs with journalctl and human-readable text files for simple tools like grep and tail.

This article provides a good overview of the most important log files and their purposes.

2. The Traditional Approach: Exploring /var/log

The /var/log directory is the classic home for log files. As you saw in the reading, it contains logs for system events, authentication, package management, and specific applications. To investigate these, we use standard text-manipulation tools.

Let's watch a video that walks through the most important files in /var/log and demonstrates how to view them.

Practice Lab: Real-Time Troubleshooting

Let's replicate the most powerful technique shown in the video: monitoring a log file in real-time. This is a go-to method for diagnosing login issues.

1. Open two terminals.

2. In Terminal 1, start following the authentication log. On Debian/Ubuntu systems, this is /var/log/auth.log. On RHEL/Fedora/CentOS, it's /var/log/secure.

   
   
   
   # On Debian/Ubuntu
   tail -f /var/log/auth.log
   
   
   
   
   # On RHEL/Fedora/CentOS (you might need sudo)
   sudo tail -f /var/log/secure
   

   Your terminal will now display the last few lines of the file and wait for new entries.

3. In Terminal 2, attempt to SSH into your own machine as a non-existent user.

   ssh somefakeuser@localhost
   

4. Switch back to Terminal 1. You should instantly see new lines appear, logging the failed authentication attempt. This immediate feedback is invaluable during troubleshooting.

Press Ctrl+C in Terminal 1 to stop tail.

3. The Modern Approach: The journalctl Swiss Army Knife

While tail and grep are great for simple text files, journalctl is the superior tool for navigating the modern systemd journal. Because the journal is structured, journalctl allows for powerful filtering that isn't possible with plain text files.

Let's explore journalctl with a video dedicated to its core functions.

Practice Lab: Diagnosing a Service Failure

This is the bread-and-butter of a system administrator. A service is failing, and you need to know why.

1. Let's check the status of a common service, sshd.

   systemctl status sshd
   

   Notice that the output includes the most recent log entries for this service. This is your first clue.

2. Now, let's say the service was failing and you needed more history. You can use journalctl with the -u (unit) flag to see all logs just for that service.

   journalctl -u sshd.service
   

   Use the arrow keys to scroll, and press q to quit.

3. This can still be a lot of information. Let's narrow it down to only show logs from the last hour and only show errors or worse. The -p flag stands for priority.

   journalctl -u sshd.service --since "1 hour ago" -p err
   

   This command is extremely powerful. In a real incident, it cuts through thousands of informational messages to show you exactly what went wrong and when.

4. Application-Specific Logs

It's important to remember that many complex applications manage their own logs inside /var/log. A web server like Apache or Nginx is a perfect example. While systemd might log that the service started or stopped, the details of web requests (access logs) and application-level errors (error logs) are in their own files.

When troubleshooting an application, your process should be:

1. Check the service status with systemctl status <service-name>.
2. Check the systemd journal for service-level errors with journalctl -u <service-name> -p err.
3. Check the application's specific logs in /var/log/<application-name>/ for detailed error messages.

Conclusion

Great job! You've just learned one of the most fundamental skills in system administration. Knowing where to look for logs and how to filter them is the difference between a 10-minute fix and a multi-hour outage.

Key Takeaways:

• Two Logging Systems: Linux uses both the traditional /var/log text files and the modern systemd journal.
• Key Log Locations:
  • /var/log/syslog or /var/log/messages: General system activity.
  • /var/log/auth.log or /var/log/secure: Login and authentication events.
  • /var/log/<app_name>/: Application-specific logs (e.g., /var/log/apache2/).
• Essential Commands:
  • tail -f <file>: To watch a text log file in real-time.
  • grep <pattern> <file>: To search for specific text within a log.
  • journalctl: The all-in-one tool for querying the systemd journal.
  • journalctl -u <service>: The go-to command for diagnosing a specific service.
  • journalctl -p err: To filter out noise and see only errors.

In our next lesson, we will focus specifically on hardware-related issues. You'll learn how to use dmesg to inspect kernel ring buffer messages. This is the log you'll turn to when the system has a problem with a disk, a network card, or any other piece of physical hardware.