Introduction
Welcome back. In our previous lesson, we examined how KVM integrates with the host Linux scheduler, establishing that vCPUs are essentially Linux threads. We saw how the scheduler manages their execution and how mechanisms like preemption notifiers ensure a clean separation between host and guest state.
Now, we shift our focus from the CPU to the second pillar of virtualization: memory. A guest operating system expects to see a contiguous physical address space starting from zero, but it's actually running on top of a host OS that manages the real machine memory. How does KVM bridge this gap?
This lesson addresses precisely that question. Our learning outcome is to describe how KVM utilizes hardware support (EPT/NPT) for efficient guest memory virtualization. We'll start by looking at the older, software-only method to understand the problem that hardware assistance was designed to solve. Then, we will dive deep into the mechanism of hardware-assisted memory virtualization and its performance implications.
By the end of this lesson, you will be able to explain:
- The fundamental challenge of memory virtualization: translating guest addresses to host addresses.
- The difference between software-based "shadow paging" and hardware-assisted "nested paging."
- How Extended Page Tables (EPT) and Nested Page Tables (NPT) work at a hardware level.
- How KVM leverages these features and handles events like EPT violations.
- The performance trade-offs involved and the importance of large pages.
First, let's establish some common terminology from the KVM documentation that we will use throughout this lesson.
The x86 kvm shadow mmu - Acronyms
The Linux kernel documentation for the KVM MMU provides a standard set of acronyms for discussing memory virtualization. Familiarizing yourself with these will make understanding the concepts much easier.
Review the 'Acronyms' section. The most important terms for us today are: gva: guest virtual address gpa: guest physical address hpa: host physical address tdp: two dimensional paging (the vendor-neutral term for EPT and NPT)
1. The Challenge: From Guest Physical to Host Physical
A guest OS manages its own page tables, which perform the translation from a Guest Virtual Address (GVA) to what it believes is a Guest Physical Address (GPA). However, this GPA is not the final machine address. The hypervisor must perform a second translation from the GPA to a Host Physical Address (HPA), which is the actual location in the machine's RAM.
GVA -> GPA -> HPA
The core challenge of memory virtualization is performing this two-stage translation efficiently.
2. The Software-Only Approach: Shadow Paging
Before direct hardware support was available, hypervisors used a technique called shadow paging.
In this model, the hypervisor (KVM) creates and maintains a "shadow" set of page tables. These shadow tables map Guest Virtual Addresses (GVAs) directly to Host Physical Addresses (HPAs). The hardware MMU is pointed to these shadow tables, not the guest's own page tables.
This diagram from a VMware paper illustrates shadow paging. The VMM (hypervisor) creates shadow page tables that map logical (guest virtual) pages directly to machine (host physical) pages, bypassing the guest's own page tables for hardware lookups.
The Problem with Shadow Paging:
The guest OS is unaware of this. When it tries to modify its own page tables (e.g., creating a new mapping, changing permissions), the hypervisor must intercept this action. It does this by write-protecting the guest's page tables. Any attempt by the guest to write to them causes a VM exit. KVM then emulates the instruction, updates its internal state, and synchronizes the change with the shadow page tables.
This frequent trapping and synchronization create significant overhead, especially for workloads that heavily manipulate memory, such as databases or applications that frequently create and destroy processes.
Performance Evaluation of Intel EPT Hardware Assist
This document from VMware, though specific to their hypervisor, provides an excellent and clear explanation of the concepts behind software and hardware MMU virtualization that apply universally.
Read the sections 'Background' and 'Software MMU' (on page 2). Focus on understanding how the VMM must 'intercept guest page table updates to keep the shadow page tables coherent with the guest page tables' and why this 'caused some overhead'.
3. The Hardware Solution: EPT and NPT
To eliminate the overhead of shadow paging, Intel introduced Extended Page Tables (EPT) and AMD introduced Nested Page Tables (NPT). KVM refers to this technology by the vendor-neutral term Two-Dimensional Paging (TDP).
With TDP, the CPU's MMU itself becomes capable of performing the two-stage address translation in hardware.
- Stage 1 (GVA -> GPA): The hardware walks the guest's page tables (pointed to by the guest's CR3 register) to translate the GVA to a GPA. This happens just like it would on a native system, without VM exits.
- Stage 2 (GPA -> HPA): The hardware then takes the resulting GPA and walks a second set of page tables, the EPT/NPT tables, to translate it to the final HPA. KVM creates and manages these tables and tells the processor where to find them by loading a pointer into a special control register (the EPTP).
This process completely removes the need for KVM to trap guest page table modifications. The guest can manage its page tables freely, and the hardware handles the full translation.
Let's study the mechanism in more detail.
Performance Evaluation of Intel EPT Hardware Assist
The same VMware document gives a concise overview of how the hardware MMU works with EPT.
Now, read the 'Hardware MMU' section on page 3. Pay close attention to the description of the two-level walk: 'for every PPN [GPA] accessed during the guest page table walk, the hardware also walks the nested page tables to determine the corresponding MPN [HPA]'.
The diagram below illustrates the EPT page walk. Notice its similarity to a standard x86-64 page walk. The GPA is broken into parts, with each part serving as an index into a different level of the EPT hierarchy until the final HPA is found.
4. KVM's Role: Managing EPT and Handling Faults
When using TDP, KVM's primary responsibility shifts from trapping guest behavior to setting up the necessary data structures for the hardware. KVM's MMU code operates in what the kernel documentation calls "direct mode."
- KVM allocates memory for the guest from the host.
- It builds the EPT/NPT tables that map the GPA range allocated to the guest to the actual HPA locations.
- It loads the EPT Pointer (EPTP) into the VMCS before a VM entry.
What happens if a mapping isn't in the EPT table?
This is analogous to a normal page fault. If the hardware walks the EPT tables for a given GPA and can't find a valid mapping, it triggers a specific type of VM exit called an EPT Violation (or NPT Fault).
Control returns to KVM. KVM's EPT violation handler then inspects the fault. A common cause is that the guest is accessing a piece of its physical memory for the first time. In response, KVM can update its EPT tables to create the GPA->HPA mapping and then resume the guest. The guest is completely unaware that this fault and resolution occurred.
The image below shows what an EPT violation looks like in a debugger, capturing the moment the hardware exits to the hypervisor.
5. Performance Implications and Large Pages
The Good: TDP dramatically reduces the number of VM exits, providing a significant performance boost for most workloads.
The Trade-off: The cost of a TLB miss is now higher. On a native system, a TLB miss results in one page walk. With TDP, a TLB miss for a GVA can result in two page walks (one for GVA->GPA, and another for GPA->HPA), potentially accessing many levels of page tables in memory.
The Solution: Large Pages
This increased TLB miss penalty is why large pages are critical for virtualization performance. A standard 4KB page requires one TLB entry. A 2MB large page also requires one TLB entry but covers 512 times more memory.
By using large pages in both the guest OS and, crucially, in KVM's EPT tables, a hypervisor can drastically reduce the TLB miss rate. This mitigates the cost of the two-stage translation and is key to achieving near-native performance.
The x86 kvm shadow mmu - Large Pages
The KVM documentation discusses its support for large pages and the conditions required to use them. This reinforces their importance from the kernel developer's perspective.
Read the short section 'Large pages'. Note the key constraint: 'the guest pte must be a large pte of at least equivalent size (if tdp is enabled, there is no guest pte and this condition is satisfied)'. This highlights that when using TDP, KVM can create large GPA->HPA mappings even if the guest isn't using large pages internally, although performance is best when both do.
An interview question might be framed like this: "You've moved a workload into a KVM virtual machine and observe high CPU usage and poor performance. Profiling with perf shows an unusually high rate of dTLB-load-misses. What could be the cause and how would you address it?"
Your answer should explain the two-dimensional page walk introduced by EPT/NPT, identify it as the likely cause of increased TLB pressure, and propose enabling large page support (e.g., Transparent Huge Pages on the host and in the guest) as the primary solution.
Conclusion
In this lesson, we demystified how KVM provides efficient memory virtualization. We saw that while older software methods like shadow paging were complex and slow, modern hardware has taken over the heavy lifting.
Key Takeaways:
- Shadow Paging: The legacy software-only method that maps GVA directly to HPA but incurs high overhead from trapping guest page table modifications.
- EPT/NPT (TDP): The hardware-assisted method where the CPU performs a two-stage address translation (GVA -> GPA -> HPA) in hardware, eliminating most memory-related VM exits.
- KVM's Role with EPT: KVM is responsible for building and managing the EPT/NPT tables that define the GPA -> HPA mapping for the guest.
- EPT Violations: These are VM exits triggered by the hardware when a GPA->HPA mapping is missing. KVM handles these faults, typically by creating the mapping on demand.
- Performance Trade-off: EPT reduces VM exits but increases the cost of a TLB miss. Using large pages is the critical optimization to mitigate this cost.
Preview of the Next Lesson:
We have now covered how KVM virtualizes the CPU and memory. The final piece of core system architecture is I/O. In our next lesson, we will explain the I/O interception process in KVM and how it enables emulated device access. We will explore how KVM traps access to device memory and I/O ports, and how QEMU uses this to emulate devices like network cards and disk controllers for the guest.