Memory-Mapped I/O for Large Datasets: Performance Advantages

Welcome to the third module of our course. In the previous module, you transformed the computational core of our vector search engine, achieving significant speedups by harnessing the power of SIMD instructions. That was a deep dive into CPU-bound optimization. Now, we turn our attention to another critical performance bottleneck: Input/Output (I/O). A high-performance search engine must be able to load, save, and access massive datasets from disk without being crippled by I/O latency.

This lesson begins our exploration of efficient I/O. Your objective is to explain the performance benefits of memory-mapped I/O (mmap) over standard read/write calls for large datasets. We will investigate why directly mapping a file into memory can be drastically faster than the conventional approach of reading it chunk by chunk. This foundational understanding is crucial before we begin implementing our file storage layer in the lessons to come.

The Overheads of Traditional File I/O

When you use a function like read() to access data from a file, it seems simple from a programmer's perspective. However, behind the scenes, the operating system is performing a series of steps that incur significant overhead.

System Call Overhead: Each call to read() is a system call, which requires a context switch from your application's unprivileged user mode to the OS's privileged kernel mode. This switch isn't free; the CPU must save the state of your program (registers, etc.), execute the kernel code, and then restore your program's state to return the data. For many small reads, this overhead accumulates quickly.
Data Copying: When the kernel executes the read() call, it first loads the required data from the storage device into an internal buffer in kernel memory called the page cache. Then, to deliver this data to your application, the kernel must copy it from the page cache into the buffer you provided in your read() call. This copy from kernel space to user space is a memory-to-memory operation that consumes CPU cycles and memory bandwidth.

This process essentially involves two copies: one from disk to the kernel's page cache, and a second from the page cache to your application's memory.

To solidify these concepts, the following article provides an excellent background on the mechanics of system calls, page faults, and the page cache.

Why mmap is faster than system calls

This article by Alexandra (Sasha) Fedorova provides a clear, low-level explanation of the OS concepts that underpin file I/O performance.

Please read the Background section. Focus on the descriptions of System calls, understanding the cost of crossing the user-kernel boundary. Also, read the brief explanations of Page faults and the Buffer cache.

The `mmap` Paradigm: A Direct Window to the File

Memory-mapped I/O takes a fundamentally different approach. Instead of asking the kernel to read data for you, you ask it to map a file (or a portion of it) directly into your process's virtual address space.

What does this mean? The OS creates a mapping in your process's page table that links a range of virtual addresses to the contents of the file on disk. After this one-time setup call to mmap(), your program can access the file's contents as if it were a giant array in memory, using simple pointer arithmetic instead of read() and write() calls.

This video provides a great conceptual overview of how mmap uses the virtual memory system.

understanding mmap, the workhorse behind keeping memory access efficient in linux

In this video from his systems programming course, Chris Kanich explains how memory mapping leverages the indirection provided by page tables.

First, watch the introduction to understand the core idea of page tables as indirection. Then, skip ahead to the segment comparing read/write with mmap, paying close attention to the discussion on reducing system call overhead.

How `mmap` Delivers Performance

By mapping the file directly to memory, mmap sidesteps the primary overheads of traditional I/O.

Reduced System Calls: You issue one mmap() system call to set up the mapping. From that point on, all access to the file data happens through direct memory access. There are no more per-read context switches from user mode to kernel mode. Your code simply dereferences a pointer, and the CPU's Memory Management Unit (MMU) handles the address translation.
Elimination of a Memory Copy: This is the most significant benefit. Since your process's virtual address space now points directly to the kernel's page cache, there is no need for the kernel to copy data from its buffer to your application's buffer. When you access a memory address in the mapped region, the hardware can access the data directly from the page cache. This avoids an entire memory-to-memory copy operation, saving CPU cycles and reducing memory bus contention.

The Role of Page Faults

This might sound like magic, but it's a clever use of the virtual memory system's existing machinery. When your program tries to access a virtual address in the mapped region for the first time, the MMU sees there's no valid mapping to physical RAM yet. This triggers a page fault, which transfers control to the OS.

The OS's page fault handler then checks its records. It sees that this address corresponds to a memory-mapped file.

If the required file page is already in the page cache (a "warm" cache), the OS simply updates the process's page table to point to that physical memory location and resumes the program. This is a very fast operation called a minor fault.
If the page is not in the page cache (a "cold" cache), the OS must first read the data from the disk into the page cache, then update the page table, and then resume the program. This is a major fault and is much slower due to the disk I/O.

The sequence diagram below illustrates the fast path, where a page fault occurs but the data is already in the page cache.

This diagram shows the sequence for a minor page fault. The application's access triggers a fault, but since the data is already in the RAM page cache, the OS only needs to look it up and update the page table before resuming the application. No disk I/O is required.

Quantifying the Gains

The theoretical benefits are clear, but the real-world performance difference can be dramatic, especially for workloads where data is accessed repeatedly (leading to a warm page cache).

The benchmark results in the chart below compare the throughput of reading a file sequentially using standard read() system calls versus mmap().

This chart compares the read throughput for `readsyscall` (purple) and `readmmap` (yellow) on a warm cache. Across all block sizes, `mmap` achieves significantly higher throughput, demonstrating its efficiency.

This isn't just a minor improvement; in this test, mmap is more than twice as fast. To understand precisely where this massive gain comes from, we can look at the CPU's activity during the benchmark. The following article does exactly that.

Why mmap is faster than system calls

This is the same article as before. Now we'll dig into the experimental results and the core explanation.

Start by reading Why mmap should be faster, which summarizes the two main reasons we've discussed. Then, review The results section to see the experimental setup and outcome. Finally, and most importantly, read the Explanation section (breakdown). Pay close attention to the CPU profiles. Notice that the read() test spends most of its time in copy_user_enhanced_fast_string, while the mmap test spends its time in __memmove_avx_unaligned_erms. This is the crux of the matter: mmap allows the data copy (if one is needed by the application) to happen in user space using a highly optimized function that leverages SIMD (AVX) instructions, whereas the kernel's copy function is more generic and conservative. This is a perfect example of how avoiding the kernel can unlock further performance.

The key insight from the article is that even when an application does copy data out of the mapped region, mmap is still faster because user-space memcpy can be more aggressive with CPU-specific optimizations (like the AVX instructions you've already used) than the kernel's copy_to_user function can be.

A More Nuanced View

While mmap is a powerful tool, it is not a silver bullet. For certain access patterns, its overheads can negate its benefits. The primary cost is in page table management and page fault handling. When you map a very large file, the OS has to prepare to manage a huge number of potential page faults.

The following research paper provides an excellent analysis of these costs, especially in the context of modern, fast storage.

Efficient Memory Mapped File I/O for In-Memory File Systems

This paper from USENIX HotStorage '17 analyzes the overheads of mmap on fast NVM storage.

In the PDF, navigate to Section 2, "Motivation". Read the analysis presented in the two paragraphs following Figure 1, starting from this paragraph. The authors show a scenario where default-mmap is actually slower than a standard read system call because of the high cost of handling a page fault for every single access. This highlights that mmap's performance is highly dependent on the access pattern and the state of the page cache.

This nuanced understanding is vital for a systems engineer. Knowing when and why a tool is effective is just as important as knowing how to use it. For our vector database, where we will be scanning large, contiguous chunks of data, the benefits of mmap will far outweigh its costs.

Conclusion

In this lesson, you've explored the fundamental reasons mmap outperforms traditional I/O for large-scale data access. You've moved beyond a superficial understanding and delved into the low-level mechanics of context switches, memory copies, and page faults.

Key Takeaways:

Traditional read() calls incur overhead from repeated system calls (user/kernel context switches) and an explicit memory copy from the kernel's page cache to your application's buffer.
mmap() avoids these overheads by mapping a file directly into the process's virtual address space.
The primary performance gains come from eliminating the kernel-to-user memory copy and reducing system calls to a single setup operation.
Access to mapped memory is mediated by the CPU's MMU and the OS's page fault handler, which efficiently brings data from disk into the page cache on demand.
The performance of mmap is at its best with a warm cache, where data is already present in RAM, turning expensive disk reads into cheap page table updates.

You now have the conceptual framework needed to work with memory-mapped files. In the next lesson, we will put this theory into practice. You will learn how to use the POSIX APIs to create, pre-allocate, and map a file into memory, laying the physical storage foundation for our vector index.

Recommended resources