Diagnosing Linux Storage Bottlenecks with System Evidence

Hello. Last lesson, you distinguished memory pressure from a merely full page cache and noted that reclaim or swapping can generate storage I/O. Now the diagnostic question changes: is storage itself delaying the workload, or is a service failing because the filesystem cannot accept another file or open handle? Both can look like “a disk problem,” but they require different evidence.

In about 40 minutes, you’ll learn to connect an affected path to its filesystem and device, interpret latency, throughput, and queue measurements, and check capacity, inodes, and file descriptors before choosing a response.

---

Locate the storage the service actually uses

Start with the symptom and its path. A slow database read, a failed log write, and a failure to create a temporary file need not involve the same filesystem. Linux presents one directory tree, but different paths can reside on different mounts; a container path may also be backed by a host mount that is not obvious from inside the container.

The first part of this video establishes how to inspect that layout. Watch the commands, not the demonstration’s mount and unmount operations—you do not need to change mounts while diagnosing a live service.

For an affected path, such as /var/lib/app/data, gather a read-only snapshot:

findmnt -T /var/lib/app/data -o TARGET,SOURCE,FSTYPE,OPTIONS
lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINTS
df -hT /var/lib/app/data
df -i /var/lib/app/data

findmnt -T identifies the mount containing the path; lsblk helps relate block devices, partitions, and mounts. df -hT checks filesystem data-block capacity, while df -i checks inode availability. Record the mount target and source before investigating a device: the busiest disk on the host may have nothing to do with the failing path. Some network filesystems and container-backed paths will not correspond neatly to one local device in iostat.

---

Separate incoming work from time spent waiting

A storage service has two different performance questions. Throughput asks how much data it moves per second; latency asks how long an operation takes. The workload matters too: thousands of small random operations can create pressure at modest bandwidth, whereas a large sequential transfer can move substantial bandwidth with relatively few operations.

Read the selected portions of this guide for a compact introduction to the device and process views. Its examples are useful, but treat claims that a particular %util or CPU I/O-wait value proves saturation as heuristics, not rules.

On a host with iostat installed, sample several intervals during the symptom:

iostat -xz 1 6

The first report typically covers time since boot; use the subsequent one-second reports to judge what is happening now. Depending on the sysstat version, column names and displayed units vary. These are the relationships to look for:

• r/s and w/s describe read and write operations per second; read/write kilobytes per second describe bandwidth. Both characterize demand.
• r_await, w_await, or await describe average block-request time, usually in milliseconds, including time spent waiting and being serviced at the measured layer. They are not the application’s entire file-operation latency.
• aqu-sz is the average number of requests outstanding, including in-flight work; it is not purely the number waiting in a software queue. A sustained rise alongside rising await suggests requests are accumulating.
• %util indicates the fraction of the sample interval with I/O in progress. Near 100% is strong evidence of busyness, but not a universal capacity ceiling: SSDs and virtualized devices can service concurrent requests.

Compare these values with the affected service’s latency and a known-good period. If 4 KiB requests arrive at 5,000 operations per second, they account for only about 20 MiB/s; low-looking bandwidth does not mean low demand for operations. Conversely, high bandwidth with stable, acceptable latency can be healthy.

CPU %iowait and load average are supporting clues, not verdicts. As in the CPU lesson, load can include tasks blocked in uninterruptible I/O. A high %iowait tells you CPUs spent idle time with I/O outstanding; it neither identifies the affected device nor measures how long the application waited. Low %iowait also cannot clear storage as a suspect on an otherwise busy CPU.

---

Understand what a device average can—and cannot—locate

The block layer receives requests, may hold or combine them, dispatches them to a driver, and records completion. The supplied diagram gives you a useful map for asking where delay might occur.

One important qualification: an application’s buffered write() can return after changing page-cache contents, before a corresponding block write completes. An application request need not map one-to-one to a block request. Thus the diagram’s block-level timing is not an end-to-end timing of every application call, and iostat’s await cannot by itself say whether delay arose before dispatch or during device service.

If the distinction matters, Red Hat’s blktrace discussion introduces queue (Q), dispatch (D), and completion (C) events. This is a deeper investigation after ordinary device and process measurements have established a problem, not a tool to run reflexively on every host.

To move from “this device is slow” to “what work is driving it,” sample processes in the same incident window:

pidstat -d 1 5
sudo iotop -boP -n 5

These tools may need installation or elevated access. pidstat provides per-process read/write rates; iotop provides a live attribution view. Check whether the process is generating excessive I/O or is itself suffering from slow I/O. Writeback and filesystem journal threads can appear as writers even when an application produced the original changes; buffered writes can also make immediate process-to-device attribution imperfect. Correlate the process view with the affected path, timestamps, and device metrics before naming a culprit.

On a cloud VM, even convincing block-level saturation does not automatically prove the physical disk is defective. The backing volume or instance can have IOPS or throughput constraints. Keep the host measurements as evidence, then compare them with the applicable cloud-volume and instance metrics.

---

Diagnose “no space” without assuming the disk is busy

A service that cannot create a file may have normal await and almost no queued I/O. That is still a storage failure, but it is a capacity or filesystem-resource problem rather than a demonstrated performance bottleneck.

Watch two short investigations from the same video. The first shows why free data space does not guarantee the ability to create a file; the second shows why deleting a named file might not free the space reported by df.

An inode represents a filesystem object’s metadata. On filesystems with a finite available inode count, many small files can exhaust inodes while df -h still shows space. The error “No space left on device” does not tell you which resource ran out. Check df -hT and df -i for the affected mount. Inode allocation behavior varies by filesystem, so measure rather than assume that every filesystem has the same fixed inode pool.

If data-block use is high, use du to find visible usage on that filesystem. For example, after confirming that /var is the mount you intend to inspect:

sudo du -xhd1 /var

-x avoids crossing onto other filesystems. Scan a suitable subtree rather than the whole host if the directory tree is large: du walks files and can add load. If df reports substantial used space that du cannot account for, one possibility is an unlinked file held open by a process:

sudo lsof +L1

Look for the affected filesystem and a credible file size, then verify the owning process. df counts allocated blocks still held by an open file; du cannot find that file by walking directory names. Filesystem metadata, snapshots, reserved space, and the scope of your du scan can also create differences—a mismatch alone does not prove an open-file leak. Arrange a safe application-level close, log rotation, or coordinated restart if an open file is confirmed. Do not force-kill a production process merely to reclaim blocks.

---

Check file handles and filesystem errors as separate limits

An inode shortage prevents allocation of filesystem objects. A file-descriptor shortage prevents a process from opening another handle—even if the filesystem has ample blocks and inodes. Handles include sockets and pipes, not just files on disk. For a service reporting “Too many open files,” inspect its PID and limits:

ls -U /proc/1234/fd | wc -l
grep -i 'open files' /proc/1234/limits
sudo lsof -p 1234

Replace 1234 with the affected PID. The /proc listing is a changing snapshot; compare its count with the process’s soft limit and its error timestamps. A per-process limit reached by that service calls for a different investigation from a filesystem that is out of inodes. Avoid raising a limit blindly if the handle count is growing because handles are not being closed.

Finally, inspect mount state and kernel errors when writes fail despite apparent capacity:

findmnt -T /var/lib/app/data -o TARGET,SOURCE,FSTYPE,OPTIONS
journalctl -k -b --since "30 minutes ago"

A read-only mount or storage I/O errors can explain a failure that df cannot. In the final video segment, notice the warning against treating an unexpected read-only transition as a setting to reverse without finding its cause.

An administrator may intentionally mount a filesystem read-only; a filesystem may also protect itself after errors. Correlate the mount option with the service error and kernel messages. Do not remount a filesystem read-write or run a repair against an active production volume simply because writes are failing.

---

Put the evidence together

Suppose a service’s write latency rises while its filesystem still has free blocks and inodes. During the same minutes, the relevant device’s w_await and aqu-sz rise, write demand increases, and pidstat identifies an unusually active writer. That supports a write-I/O bottleneck associated with the workload. It does not yet tell you whether the limiting point is the filesystem, block queue, virtual volume, or backing service.

Now suppose the service instead fails to create new small files. iostat remains quiet, df -h shows free capacity, and df -i shows no free inodes on the affected mount. That supports inode exhaustion, not a slow disk. If both df views show room but the service reports “Too many open files,” investigate its descriptors and limits instead.

The habit is to keep six questions distinct: Which path and mount? How much work is arriving? How long does it wait? Is work accumulating? Can the filesystem allocate space and inodes? Can the process open another handle? A diagnosis is strongest when measurements and errors agree in the same time window.

---

Key takeaways

Use the affected path to identify the right filesystem and device. Pair throughput and operation rate with latency and outstanding requests; do not call a device saturated from %util, CPU I/O wait, or load average alone. For failed writes or file creation, check blocks, inodes, deleted-but-open files, descriptor limits, mount state, and kernel errors before changing anything.

Next, you’ll apply the same evidence-first approach to a failed Linux service startup, following systemd state and dependencies through to the process exit and journal messages.