Troubleshoot Resource Bottlenecks with Monitoring Tools

A Methodical Approach to Troubleshooting

When a system is slow, it's tempting to start running commands randomly. A better approach is to be systematic. A widely used methodology in performance analysis is the USE Method:

• Utilization: For a given resource (like a CPU or disk), what percentage of its capacity is being used?
• Saturation: Is the resource so busy that work is queuing up, waiting to be processed?
• Errors: Are there any explicit errors occurring?

Throughout this lesson, we'll apply this thinking to the three pillars of system performance: CPU, Memory, and I/O.

1. Diagnosing High CPU Usage

A CPU-bound system is one where the processor is the limiting factor. Programs are ready to run, but they have to wait for their turn on the CPU.

Key Tools and Metrics

1. uptime and Load Average:
   This command gives you a quick snapshot of system health, including the load average.

   uptime
   # 10:30:01 up 5 days, 16:21,  1 user,  load average: 1.50, 0.80, 0.65
   

   The three numbers represent the average number of processes in the run queue (running or waiting to run) over the last 1, 5, and 15 minutes.

   The critical insight is to compare the load average to the number of CPU cores on your system. You can find your core count with nproc.

   • If load average < number of cores: The system is handling the load.
   • If load average > number of cores: The system is saturated. Work is queuing up faster than the CPUs can process it.

2. top - The System Dashboard:
   top provides a real-time, dynamic view of the system. The summary at the top provides crucial clues:

   • load average: The same values as uptime.
   • %Cpu(s): A breakdown of what the CPU is spending its time on.
     • us: user space — time spent running user applications.
     • sy: system space (kernel) — time spent on kernel tasks, like handling I/O or system calls.
     • id: idle — time the CPU has nothing to do.
     • wa: I/O wait — time the CPU is idle but waiting for a disk or network operation to complete.

A high %us or %sy value, combined with low %id, points to a CPU bottleneck.

Let's watch a practical demonstration of identifying a CPU-bound system.

3. htop - A Friendlier top
   While top is standard, htop is an popular alternative that many find more intuitive. It provides the same information in a color-coded, easier-to-read format with a few extra features. It is not installed by default on all systems but is usually available in the main repositories.

2. Diagnosing Disk I/O Issues

An I/O-bound system is one where performance is limited by the speed of its disks or network interfaces. The CPU is often idle because it's waiting for data to be read from or written to a storage device.

Key Tools and Metrics

The first clue often comes from the %wa (I/O wait) value in top or vmstat. If this number is consistently high, your CPUs are spending a lot of time waiting on I/O.

1. iostat - I/O Statistics:
   This is the primary tool for drilling into disk performance. Running iostat -x 1 will give you extended statistics, updated every second.

   $ iostat -x 1
   Device: ... r/s   w/s   rkB/s   wkB/s ... await  %util
   sda       ... 2.45  8.67  45.23  156.89 ... 7.23   2.8
   

   Key columns to watch:

   • %util: The percentage of time the disk was busy. If this is consistently above 80-90%, the disk is saturated.
   • await: The average time (in milliseconds) an I/O request waits for completion. High values indicate slow disk response.
   • avgqu-sz: The average queue length. A high number means I/O requests are piling up, waiting to be serviced by the disk.

2. ps and Uninterruptible Sleep (D state):
   Sometimes, a process becomes "stuck" waiting for an I/O operation that will never complete (e.g., waiting on a disconnected network drive). These processes enter an uninterruptible sleep state, marked with a D in the STAT column of ps aux. A large number of processes in the D state is a strong indicator of a severe I/O problem.

The next video segment demonstrates how to track down a tricky I/O issue caused by a failed network mount.

3. Diagnosing High Memory Usage

A system experiences memory pressure when it doesn't have enough physical RAM to hold all the active programs and data. To compensate, the kernel starts moving less-used memory pages to a dedicated disk area called swap space. Because disks are orders of magnitude slower than RAM, heavy swapping can bring a system to a crawl.

Key Tools and Metrics

1. free -h - Memory Overview:
   This is the quickest way to check your memory status.

   $ free -h
                 total        used        free      shared  buff/cache   available
   Mem:           7.7Gi       2.1Gi       3.7Gi       1.0Mi       1.9Gi       5.3Gi
   Swap:          2.0Gi          0B       2.0Gi
   

   • free vs. available: Don't be alarmed by a low free value. Linux uses spare RAM for disk caching (buff/cache) to speed things up. The available column gives a more realistic estimate of how much memory can be made available for new applications without swapping.
   • Swap: Watch the used value here. If it's consistently high or increasing, your system is actively swapping.

2. vmstat - Virtual Memory Statistics:
   Running vmstat 1 gives you a second-by-second update on memory activity.

   $ vmstat 1
   procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
    r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
    2  0      0 1843200  80532 947012    0    0     5    23   45   78  3  1 96  0  0
   

   The swap columns are critical:

   • si (swap in): Memory being read from swap back into RAM.
   • so (swap out): Memory being written from RAM to swap.
     Any sustained, non-zero activity in si or so indicates memory pressure.

3. The OOM Killer:
   In extreme cases of memory exhaustion, the kernel invokes the Out-Of-Memory (OOM) Killer. It chooses a process (based on a heuristic score) and kills it to free up memory. You will see messages about this in the kernel logs, which you can view with dmesg or journalctl -k. If you see OOM killer messages, your system is under severe memory distress.

This final video segment covers these memory monitoring tools and the OOM killer.

A Practical Diagnostic Workflow

You now have tools for each subsystem. The real skill is putting them together. Here's a logical workflow to follow when a system is slow:

1. Overall Health Check (uptime): Is the load average high compared to the number of CPUs? This is your first and fastest clue.
2. Find the Bottleneck (top/htop):
   • High %us or %sy? It's likely a CPU issue.
   • High %wa? It's likely a disk I/O issue.
   • Low on available memory in the summary? It could be a memory issue.
3. Drill Down:
   • For I/O issues: Use iostat -x 1 to check disk utilization and latency. Use ps aux to look for processes in D state.
   • For Memory issues: Use free -h to confirm memory status and vmstat 1 to check for active swapping (si/so). Check dmesg for OOM killer messages.

This diagram provides a great mental map of the various tools and which part of the Linux system they help you investigate.

Conclusion

In this lesson, we've established a methodical foundation for troubleshooting system performance. You learned how to distinguish between CPU, memory, and I/O bottlenecks and the primary commands to diagnose each.

Key Takeaways:

• A systematic approach (like the USE method) is crucial for effective troubleshooting.
• CPU Issues: Identified by high load average (uptime) and high %us/%sy in top.
• Disk I/O Issues: Indicated by high %wa (I/O wait) in top and confirmed with high %util or await in iostat. Processes stuck in D state are a red flag.
• Memory Issues: Diagnosed by observing low available memory, active swapping (si/so in vmstat), or OOM killer messages in dmesg.
• The standard workflow is to start broad with uptime and top, then use specialized tools like iostat and vmstat to confirm your hypothesis.

Preview of the Next Lesson:
We've just learned how to troubleshoot a running system. But what happens when the system doesn't even start properly? In our next lesson, we will identify common boot failure stages and symptoms, preparing you to diagnose and fix problems that occur before you can even get to a login prompt.