KVM Architecture and Hardware Virtualization

1. The KVM and QEMU Partnership

A common point of confusion is the relationship between KVM and QEMU. They are not the same thing, but rather two components that work together to provide a full virtualization solution. KVM provides the core CPU and memory virtualization, while QEMU provides everything else.

• KVM (Kernel-based Virtual Machine): A Linux kernel module (kvm.ko, plus kvm-intel.ko or kvm-amd.ko). It is a "Type-1" or "native" hypervisor. Its primary job is to use the hardware virtualization extensions (VT-x/AMD-V) we discussed previously. It does not emulate hardware like disk drives or network cards. Instead, it exposes the hardware's virtualization capabilities to user-space through a character device file: /dev/kvm.

• QEMU (Quick EMUlator): A powerful user-space machine emulator and virtualizer. QEMU can emulate a full system (CPU, memory, and peripherals) entirely in software. However, when used with KVM, it changes its role. It offloads the CPU-intensive work to KVM for near-native performance and focuses on emulating the other machine hardware (e.g., a storage controller, a USB bus, a network interface card).

The following video provides an excellent overview of this division of labor.

This architectural separation is key. QEMU runs as a standard Linux process. When it needs to run guest code, it doesn't emulate the CPU instructions itself; it tells KVM to do it. KVM then uses the hardware extensions to run the guest code directly on the physical CPU. When the guest needs to access a piece of emulated hardware (like writing to a virtual disk), a VM Exit occurs. KVM passes control back to the QEMU process, which handles the request, and then tells KVM to resume the guest.

2. The KVM API: A Device Driver Model

Since you have experience with virtual device drivers, the KVM interface model should feel familiar. KVM exposes its functionality through a character device, /dev/kvm. A user-space program like QEMU interacts with this device using file operations, primarily open() and ioctl().

Each core object in the KVM world—the VM itself, and each virtual CPU (vCPU)—is represented by a file descriptor.

1. System File Descriptor: Obtained by open("/dev/kvm", O_RDWR). This is the main entry point to the KVM subsystem.
2. VM File Descriptor: Created by an ioctl() on the system file descriptor. This represents an entire virtual machine, including its memory address space and other system-wide attributes.
3. vCPU File Descriptor: Created by an ioctl() on the VM file descriptor. This represents a single virtual CPU core within the VM, including its registers and execution state. A multi-core guest VM will have multiple vCPU file descriptors.

Let's watch a detailed walkthrough of this process.

The video you just watched outlined the entire lifecycle. Now, let's break it down and look at the specific API calls that make it happen.

3. A Practical Walkthrough of the KVM API

The best way to understand an API is to see how it's used in code. We'll now walk through a minimal example of a user-space program that uses KVM to run a tiny piece of code in a VM. This demonstrates the core concepts far more directly than digging through the massive QEMU codebase.

The following resource is a classic LWN.net article that builds a minimal VMM from scratch. We will use it as a guide to understand the essential ioctl calls.

The Setup Phase

As you've just read, the setup process is a methodical series of requests from user-space to the kernel:

1. Get a handle to KVM: open("/dev/kvm") gives you a file descriptor kvm.
2. Create a VM: ioctl(kvm, KVM_CREATE_VM, 0) tells the kernel to create the data structures for a new virtual machine. It returns a new file descriptor, vmfd, which represents this VM.
3. Allocate Guest Memory: The user-space process calls mmap() to allocate a block of its own memory. This memory will serve as the guest's "physical" RAM.
4. Inform KVM about the Memory: ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region) tells KVM to map this user-space memory into the guest's physical address space. This is how the guest gets its RAM.
5. Create a Virtual CPU: ioctl(vmfd, KVM_CREATE_VCPU, 0) creates a vCPU, returning yet another file descriptor, vcpufd.
6. Initialize CPU State: Using ioctls like KVM_SET_REGS and KVM_SET_SREGS on the vcpufd, the user-space process sets the initial state of the vCPU's registers (e.g., setting the instruction pointer rip to the start of the code and initializing the stack pointer rsp).

The Execution Loop

Once the VM is configured, we enter the main execution loop. This loop is the heart of any KVM-based VMM.

Let's look at the code for this loop in the LWN article.

The loop embodies the entire virtualization process:

1. ioctl(vcpufd, KVM_RUN, NULL): This is a blocking system call. The user-space thread pauses, and the kernel's KVM module takes over. KVM performs a VM Entry, and the guest code begins executing directly on the CPU.
2. Guest Execution & VM Exit: The guest runs until an event triggers a VM Exit (e.g., executing an OUT instruction). The hardware traps back into the KVM kernel module.
3. Return to User-space: KVM saves the exit reason and other relevant data into a shared memory structure (struct kvm_run) and the KVM_RUN ioctl returns. The user-space thread resumes execution.
4. Handle the Exit: The user-space code inspects run->exit_reason. If it was KVM_EXIT_IO, it emulates the I/O operation. If it was KVM_EXIT_HLT, it terminates the VM.
5. Loop: The process repeats, with user-space calling KVM_RUN again to resume the guest.

Interview Scenario: Putting It All Together

To solidify your understanding, let's tackle a typical deep-dive interview question that synthesizes these concepts.

Interviewer: "QEMU is a user-space process, which normally can't execute privileged instructions. KVM is part of the kernel. Explain, step-by-step, how QEMU uses KVM to run a guest OS kernel, which does need to execute privileged instructions. Where do the VM Entries and VM Exits fit into this process?"

Your Thought Process:

1. The Interface: The connection is the /dev/kvm character device. QEMU uses ioctls.
2. Setup: QEMU acts as the "setup" program. It creates the VM, allocates its RAM from its own process memory (mmap), and creates vCPUs.
3. The "Run" Command: The key is the KVM_RUN ioctl. This is the handover from user-space to the kernel.
4. VM Entry: Inside the kernel, KVM uses the hardware instructions (VMLAUNCH/VMRESUME) to perform the VM Entry. This is where the CPU switches to non-root mode.
5. VM Exit: A privileged instruction in the guest causes a hardware-level VM Exit. The CPU switches back to root mode, and control returns to KVM in the kernel.
6. Return to User-space: KVM communicates the reason for the exit back to QEMU by returning from the KVM_RUN ioctl.
7. Emulation: QEMU emulates the device the guest was trying to access.
8. The Loop: QEMU calls KVM_RUN again to resume the guest. This constitutes the main VMM loop.

This sequence perfectly explains how a user-space process can orchestrate the execution of a fully privileged guest OS, leveraging the kernel and underlying hardware for the heavy lifting.

Conclusion

In this lesson, we demystified the KVM architecture and its user-space API. We saw that KVM is not a monolithic hypervisor but a kernel-level toolkit that exposes hardware virtualization features to user-space through a well-defined device driver interface.

Key Takeaways:

• KVM/QEMU Partnership: KVM handles CPU/memory virtualization using hardware support, while QEMU emulates peripheral devices.
• /dev/kvm API: KVM's functionality is exposed via the /dev/kvm character device. User-space programs use file descriptors and ioctl calls to create and manage VMs (KVM_CREATE_VM), vCPUs (KVM_CREATE_VCPU), and memory (KVM_SET_USER_MEMORY_REGION).
• The KVM_RUN Loop: The core of a KVM-based VMM is the ioctl(KVM_RUN) loop. This call triggers a VM Entry to run the guest, and it returns upon a VM Exit, allowing the user-space process to handle events like I/O and then resume the guest.

Preview of the Next Lesson:

We have established that each vCPU is managed by a user-space thread (typically in QEMU) that calls KVM_RUN. But how does the host Linux system decide when to run these threads? In our next lesson, "Explain how KVM integrates with the host Linux scheduler to manage vCPU execution," we will explore this crucial aspect. You'll learn that vCPUs are, from the host's perspective, just regular threads, and we will investigate the performance implications of this design.