Troubleshooting Service Failures

1. The First Responder: Understanding Service Status

When a user reports "the website is down," your first step isn't to start rebooting things. It's to gather information. For services managed by systemd, your primary tool for this is systemctl.

This video provides an excellent hands-on review of the essential systemctl commands. It demonstrates how to check a service's status and manage its state (start, stop, etc.), which is our starting point for any diagnosis.

Dissecting systemctl status

The systemctl status <service-name> command gives you a quick, dense summary of a service's health. Let's break down the key parts you need to understand.

From that reading, here are the most important lines to interpret:

• Loaded: Tells you where the service's unit file is located and whether it's enabled (starts at boot) or disabled.
• Active: This is the crucial line.
  • active (running): The service is running correctly.
  • inactive (dead): The service is stopped.
  • active (exited): The service started and then immediately finished, which is normal for one-off tasks but an error for long-running services.
  • failed: The service tried to start but couldn't due to an error. This is our cue to start digging deeper.
• Log Snippet: At the bottom, you'll see the last few log entries related to the service. This is often your first clue about what went wrong.

2. The Detective's Logbook: journalctl

The log snippet in systemctl status is helpful, but it's rarely the whole story. To conduct a full investigation, you need to consult the system's centralized log, and the tool for that is journalctl.

This next video focuses entirely on journalctl, showing you how to go from seeing all system logs to surgically targeting the exact information you need.

Here are the essential journalctl commands for troubleshooting:

3. A Systematic Approach to Troubleshooting

Now, let's combine these tools into a methodical workflow. Randomly trying commands is inefficient; a structured approach will solve problems faster and is what distinguishes a professional from a hobbyist.

We'll simulate a common scenario: the Nginx web server is failing. You can follow these steps on your VM. If you don't have Nginx installed, you can install it with sudo apt install nginx.

The 4-Step Troubleshooting Workflow

We'll follow the systematic process outlined in the "Complete Guide to systemctl Logs" article (LINK), which you can refer back to. Specifically, the section "Systematic Service Failure Diagnosis".

Scenario: Nginx fails to start after a configuration change.

Step 1: Initial Status Check

First, we'll intentionally break the Nginx configuration.

1. Open the main Nginx configuration file: sudo nano /etc/nginx/nginx.conf
2. Find a line that ends with a semicolon (;), for example include /etc/nginx/conf.d/*.conf;.
3. Delete the semicolon at the end of the line.
4. Save the file (Ctrl+O, Enter) and exit (Ctrl+X).

Now, try to restart Nginx:

sudo systemctl restart nginx

The command will likely hang and then fail. This is your trigger. Now, begin the diagnosis.

sudo systemctl status nginx

You will see the Active: line shows failed. At the bottom, you will see a log snippet mentioning a failure to start and possibly an error related to the configuration file.

Step 2: Examine Detailed Logs

The status gave us a clue. Let's get the full story.

sudo journalctl -u nginx.service

Scroll through the output. You should see explicit error messages. Nginx is very good about its error logging and will often point you to the problem. You might see something like nginx: [emerg] unexpected "}" in /etc/nginx/nginx.conf:25 or directive is not terminated by ";". The log has confirmed a configuration syntax problem.

Step 3: Check Configuration Syntax

Most well-designed services provide a way to test their configuration files without having to start the service itself. This is much faster and safer than repeated start/fail cycles.

For Nginx, the command is nginx -t.

sudo nginx -t

This command will test the entire Nginx configuration and print out the exact file and line number where it found the syntax error. This is the smoking gun.

Resolution:

1. Go back and edit the file: sudo nano /etc/nginx/nginx.conf
2. Add the semicolon back to the line you edited.
3. Save and exit.
4. Run the test again to confirm: sudo nginx -t. It should now report syntax is ok and test is successful.
5. Now, start the service: sudo systemctl start nginx.
6. Verify with sudo systemctl status nginx. It should now be active (running).

Step 4: Analyze Dependencies

Sometimes a service fails not because of its own configuration, but because something it depends on is missing or not running.

Let's look at a common dependency issue. The systemd-analyze tool can be used to validate unit files.

Scenario: Nginx fails due to a dependency failure.

A common error is when a service's ExecStart directive points to an executable that doesn't exist. Let's simulate this.

1. First, make a copy of the nginx service file so we can edit it safely. We'll use the --full flag which creates a copy in /etc/systemd/system, overriding the default file.

   sudo systemctl edit --full nginx.service
   

2. Your text editor will open with the service file. Find the line starting with ExecStart=.
3. Change /usr/sbin/nginx to /usr/sbin/nginx-is-not-real.
4. Save and exit the editor.
5. Tell systemd to reload its configuration since we changed a unit file:

   sudo systemctl daemon-reload
   

6. Now, try to start the service:

   sudo systemctl restart nginx
   

   It will fail. Check the status: sudo systemctl status nginx. This time, you'll see a different error. Notice the line code=exited, status=203/EXEC. As you read in the article, 203/EXEC is a specific systemd error meaning it could not execute the command. This immediately tells you the problem is with the path to the executable itself.

Resolution:

1. Run sudo systemctl edit --full nginx.service again.
2. Fix the ExecStart path back to /usr/sbin/nginx.
3. Save and exit.
4. Run sudo systemctl daemon-reload and sudo systemctl start nginx. The service will now start correctly.
5. To clean up, you can remove your override file: sudo rm /etc/systemd/system/nginx.service and run daemon-reload one last time.

Another dependency issue is when a service Requires another service that isn't running. You can check this with systemctl list-dependencies <service-name>. If you see a dependency in red or with a failed state, that's your next area to investigate.

Conclusion

You have just walked through the fundamental process of diagnosing a failed service in a modern Linux system. This methodical, four-step approach—status, logs, config, dependencies—is your blueprint for tackling a huge range of common server problems.

Key Takeaways:

• Start with systemctl status <service> to get a quick overview and see if the service is in a failed state.
• Use journalctl -u <service> to dig into the detailed logs. This is where you'll find specific error messages.
• Always check the configuration using the service's built-in test command (like nginx -t) before concluding it's a deeper issue.
• Investigate dependencies with systemctl list-dependencies and by checking for systemd-specific exit codes like 203/EXEC.

In our next lesson, we will dive into one of the most common root causes you'll discover through log analysis: permission errors. You will learn to diagnose and fix file permission issues that prevent a service from starting or a user from accessing files, building directly on the skills you've developed today.