Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 62

`kmalloc` vs. `vmalloc`: Choosing the Right Allocator

Recommended resources

Hello! Welcome back to your Embedded Interview Preparation Bootcamp.

In our last lesson, we navigated the complex world of memory ordering, distinguishing between compiler and hardware barriers. We saw how these primitives are essential for ensuring correctness in concurrent code. As we concluded, controlling ordering is only half the battle; controlling allocation is the other.

Today, we shift our focus to a cornerstone of Linux kernel memory management. We will dissect two of the most fundamental allocation functions, kmalloc and vmalloc. Understanding their differences, trade-offs, and appropriate use cases is a classic interview question that probes your deep understanding of the kernel's memory architecture.

Your learning outcome for this lesson is to: Compare kmalloc (physically contiguous) vs. vmalloc (virtually contiguous) and choose the correct allocator for different driver scenarios.

1. Kernel Memory: A Tale of Two Address Spaces

Before we can compare kmalloc and vmalloc, we must first understand the memory landscape they operate in. Inside the kernel, not all virtual addresses are created equal. They fall into two primary categories which dictate the properties of the memory they point to.

To build this foundational mental model, let's watch a few segments from an excellent talk by The Linux Foundation.

Introduction to Memory Management in Linux

This video introduces the concept of virtual memory and how it's structured within the Linux kernel. It draws a clear distinction between the two types of kernel address spaces that are central to our discussion.

Please watch the following segments: Kernel Virtual Memory Intro (12:35 - 15:49): This part explains the user/kernel memory split and introduces the key concepts of 'kernel logical addresses' and 'kernel virtual addresses'. This is the most critical section. Kernel Logical Addresses (15:49 - 18:56): Focus on how these addresses have a simple, fixed mapping to physical RAM. Pay close attention to the statement that memory in this range is physically contiguous and why that's important for DMA. Kernel Virtual Addresses (21:03 - 23:11): This section describes the second type of address space, often called vmalloc space. Note that it is 'managed dynamically' and results in non-contiguous physical memory.

From the video, we have two key takeaways:

  1. Kernel Logical Addresses: This is a region of virtual memory that has a direct, linear mapping to physical RAM. A contiguous range of logical addresses corresponds to a contiguous range of physical addresses. This is the domain of kmalloc.
  2. Kernel Virtual Addresses: This is a separate region where the kernel can create arbitrary mappings from virtual to physical addresses. A contiguous range of virtual addresses here is backed by pages that can be scattered all over physical RAM. This is the domain of vmalloc.

With this understanding, let's look at the allocators themselves.

2. kmalloc: The Fast, Contiguous Workhorse

kmalloc is the most common memory allocator in the kernel. Its primary purpose is to allocate physically contiguous blocks of memory.

  • Physical Contiguity: This is the defining feature of kmalloc. The virtual address range it returns maps to an unbroken, contiguous block of physical RAM.
  • Primary Use Case: Any scenario requiring physical contiguity, the most common of which is for Direct Memory Access (DMA). Hardware devices like network cards or storage controllers often need to write to or read from a single, continuous block of memory without the help of an MMU. kmalloc provides such memory.
  • Performance: kmalloc is very fast. For small, common object sizes, it's typically a lockless, O(1) operation.

How kmalloc Works Internally

kmalloc is not a standalone allocator. It is the front-end to the kernel's slab allocator (SLUB, SLAB, or SLOB). The slab allocator maintains caches of pre-allocated, fixed-size objects. When you call kmalloc, it serves your request from the appropriate cache. If the cache is empty, the slab layer requests new pages from the underlying buddy system allocator, which is what actually manages physical pages.

This combination is highly efficient. The following video provides a concise explanation of this relationship.

GWU OS: Memory Allocation - Slab and Buddy Allocators

This video segment explains how kmalloc is implemented as a power-of-two allocator built on top of the slab and buddy systems.

Watch from 43:59 to 46:19. The key insight here is that kmalloc is essentially a collection of slab caches for various power-of-two sizes, and the buddy allocator provides the backing memory and coalescing mechanism.

Linux Kernel kmalloc Allocation Flow (GFP_ATOMIC)
This flowchart shows the fast path for a `kmalloc` call. The request is first served from a per-CPU slab cache. Only if that is empty does it fall back to the slower path involving the buddy system allocator.

3. vmalloc: For Large, Virtually Contiguous Buffers

vmalloc serves a different purpose. It allocates a block of memory that is contiguous in the virtual address space, but whose underlying physical pages are scattered.

  • Virtual Contiguity Only: The memory looks like a single, large chunk to the CPU, but the physical pages backing it can be located anywhere in RAM.
  • Primary Use Case: Allocating very large, sequential memory regions that are only ever accessed by the CPU. Since physical contiguity is not required, the kernel has a much easier time finding enough free (but scattered) pages. A classic example is loading a kernel module's code and data into memory.
  • Key Limitation: Cannot be used for most DMA operations. A DMA engine needs a single physical start address and a length, which vmalloc memory cannot provide.
  • Performance: vmalloc is significantly slower than kmalloc. It must allocate memory for the page table entries and then build the custom virtual-to-physical mappings for the requested area.

The following reading from the classic "Linux Device Drivers" book provides an excellent textual comparison.

Linux Device Drivers: Getting Hold of Memory

This chapter from 'Linux Device Drivers, 2nd Edition' gives a detailed, practical comparison between the kernel's memory allocators.

Read the section titled 'vmalloc and Friends'. It perfectly contrasts vmalloc with kmalloc and get_free_pages, explaining the differences in their address ranges and overhead. Pay attention to the line: 'The right time to call vmalloc is when you are allocating memory for a large sequential buffer that exists only in software.'

4. The Ultimate Comparison

We can now bring everything together. In an interview, you should be able to articulate these differences clearly. The following infographic is an outstanding summary.

Linux Kernel Memory Allocation: kmalloc() vs vmalloc() - Internal Architecture
This infographic provides a side-by-side comparison of kmalloc() and vmalloc(), covering their internal architecture, memory layout, performance, and key differences. It's an excellent summary of the concepts discussed.

Let's distill this into a table, which is a great format for an interview response.

Feature kmalloc vmalloc
Memory Contiguity Physically contiguous & Virtually contiguous Virtually contiguous only (physically scattered)
Primary Use Case DMA buffers, small-to-medium kernel objects Large software-only buffers (e.g., for module loading)
Performance Fast. Uses slab caches (often O(1)). Slower. Requires page table allocation and mapping.
Size Limit Smaller. Limited by available contiguous memory. Larger. Can allocate up to available system RAM.
Underlying Mech. Slab Allocator + Buddy System Page Allocator + Page Table Manipulation
Address Range Returns a kernel logical address (direct-mapped). Returns a kernel virtual address from a dedicated range.

The Modern Alternative: kvmalloc

What if you need to allocate memory that might be large, but you're not sure? Modern kernels provide kvmalloc.

Memory Allocation Guide — The Linux Kernel documentation

The official kernel documentation provides a concise guide to memory allocation. Let's look at its recommendation.

Read the section 'Selecting memory allocator'. It explains the typical use cases for kmalloc and vmalloc and introduces kvmalloc as a hybrid solution that tries kmalloc first and falls back to vmalloc if the allocation is too large.

kvmalloc is a convenient wrapper that attempts an allocation with kmalloc first. If that fails (likely due to the requested size being too large for contiguous allocation), it automatically retries with vmalloc. The crucial thing to remember is that because it might return vmalloc'ed memory, you cannot assume the memory is physically contiguous and therefore cannot use it for DMA.

5. Practical Scenarios: You're the Driver Developer

Let's apply this knowledge. For each scenario, decide which allocator is the correct choice and why.

Scenario 1: You are writing a driver for a high-speed networking card. The hardware requires a 128 KB ring buffer for packet descriptors that it will access via DMA.

Answer: kmalloc (or more specifically, a DMA-aware API like dma_alloc_coherent which often uses kmalloc internally). The key requirement is physical contiguity for DMA. kmalloc guarantees this.

Scenario 2: You need to allocate an 8 MB buffer in your driver to hold a compressed firmware image read from a file. Your code will then decompress this data in-place before writing it to a flash device piece by piece.

Answer: vmalloc. The buffer is very large, making a contiguous physical allocation unlikely to succeed and unnecessary. Only the CPU will access this memory, so virtual contiguity is sufficient. This is a classic use case for vmalloc.

Scenario 3: Your driver needs a small, 300-byte structure to store its internal state (e.g., pointers, configuration flags, a spinlock).

Answer: kzalloc (which is kmalloc followed by memset to zero). The allocation is small, so the fast and efficient kmalloc is the standard and correct choice.

Conclusion

You have now mastered one of the most fundamental topics in Linux kernel development. The choice between kmalloc and vmalloc is a direct reflection of the trade-offs between virtual and physical memory, performance, and hardware requirements.

Key Takeaways:

  • The core decision point is physical contiguity. If your memory must be physically contiguous (primarily for DMA), you must use kmalloc.
  • kmalloc is the fast, general-purpose allocator for small-to-medium sized objects, backed by the slab/buddy system.
  • vmalloc is the specialized allocator for large, software-only buffers where only virtual contiguity is needed. It is slower due to page table manipulation.
  • kvmalloc provides a convenient fallback mechanism but its result cannot be used for DMA.

Preview of the Next Lesson:

We've discussed allocating memory, but in multi-core systems, where that memory is located relative to the CPU core accessing it can have a massive performance impact. In our next lesson, we will explore another specialized memory technique: per-CPU variables. You will learn how to use them to eliminate cache-line bouncing and false sharing, dramatically improving performance in multi-core driver code.