Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 78

Virtio Offload: vhost-net/vhost-scsi Acceleration

Recommended resources

Introduction

Welcome back! In our previous lesson, we examined the virtio transport mechanism, centered on virtqueues and shared memory. We concluded that while virtio dramatically improves on full hardware emulation, it still has a performance bottleneck: every data packet must traverse from the host kernel up to the QEMU userspace process and then be copied back down to the guest. This round trip involves costly context switches and memory copies.

This lesson addresses that final bottleneck. Our goal is to explain how the vhost-net and vhost-scsi protocols accelerate virtio by offloading the data plane to the host kernel. You'll learn how this widely-used optimization bypasses QEMU for high-speed data transfer, moving I/O processing into a dedicated kernel thread for near-native performance. This is the standard high-performance configuration for KVM, making it a critical topic for performance analysis and debugging in virtualized environments.

1. The Problem: The QEMU Userspace Bottleneck

In the "pure" virtio model we've discussed, QEMU acts as the backend for the virtio device. While it doesn't emulate the low-level registers of a physical NIC, it is still responsible for mediating all data transfers. Let's break down the overhead this creates.

Introduction to VirtIO, Part 2: Vhost

To fully appreciate the problem vhost solves, it's essential to understand the sources of overhead in the standard QEMU-based virtio model. This article from the Oracle Linux Blog, 'Introduction to VirtIO, Part 2: Vhost', provides an excellent, detailed analysis of this overhead.

Please read the initial section, 'Context: VirtIO Data Plane Overhead,' including the subsections 'Limitations of Pure VirtIO' and the detailed 'Transmit Path' and 'Receive Path' walkthroughs. Focus on where context switches and VM exits/entries occur.

As the article you just read explains, the data path for a single network packet involves a costly dance between different execution contexts:

  • Transmit (Guest -> Host): The guest driver's "kick" causes a VM exit. KVM then needs to schedule the QEMU userspace thread, resulting in a context switch from kernel to userspace. QEMU processes the packet and uses a syscall to send it to the host's network stack, which is a ring transition back to the kernel.
  • Receive (Host -> Guest): The host kernel receives a packet and must wake up the QEMU process (another context switch). QEMU reads the packet, places it in the virtqueue, and then makes an ioctl syscall to KVM to inject an interrupt, which involves another VM exit/entry.

The key takeaway is that for every batch of I/O, the QEMU process must be scheduled, perform work, and make syscalls. This user-kernel-user transition is the primary source of overhead.

Virtio vs. vhost-net Network Architecture Comparison
This diagram visually contrasts the two approaches. On the left is the standard path, where data flows from the guest, through the QEMU userspace process, and into the host kernel. On the right, the vhost-net path shows data flowing directly from the guest to a 'vhost' module within the host kernel, bypassing QEMU.

2. The Solution: Offloading the Data Plane with Vhost

The vhost protocol was designed to eliminate this userspace bottleneck. It moves the entire virtio data plane—the actual processing of virtqueues—from the QEMU process into a dedicated kernel module.

  • vhost-net is the kernel module for virtio-net devices.
  • vhost-scsi is the kernel module for virtio-scsi devices.

With vhost, QEMU's role is reduced to handling the control plane: initial device setup, feature negotiation, and managing live migration. The high-frequency data operations are handled entirely within the host kernel.

Introduction to VirtIO, Part 2: Vhost

Now, let's explore how the vhost architecture achieves this. We'll return to the Oracle blog post to see the solution.

Please read the sections 'What is Vhost?' and 'Vhost Architecture', focusing on the 'Kernel-based Vhost Architecture' subsection. Pay close attention to the new Tx and Rx paths and the roles of ioeventfd and irqfd.

How Vhost Bypasses QEMU

As you saw in the reading, the vhost architecture fundamentally changes the notification pathway. Instead of QEMU acting as a middleman, KVM facilitates a more direct link between the guest and the vhost kernel thread. Two special mechanisms, implemented using eventfd, are key to this:

  1. ioeventfd (Guest-to-Host Kicks):

    • QEMU tells KVM to associate the virtio device's "doorbell" MMIO address with a specific eventfd file descriptor. This eventfd is monitored by the vhost kernel thread.
    • When the guest driver "kicks" the device by writing to the doorbell, it triggers a VM exit as usual.
    • However, KVM now sees this address is tied to an ioeventfd. Instead of waking up QEMU, KVM directly signals the eventfd.
    • This signal immediately wakes up the sleeping vhost-net kernel thread to process the available ring.
    • Result: The context switch to QEMU is completely eliminated from the transmit path.

  2. irqfd (Host-to-Guest Interrupts):

    • QEMU creates an eventfd and registers it with KVM as a virtual interrupt source for the guest (an "irqfd"). It passes this fd to the vhost module.
    • When the vhost-net thread has processed buffers and placed them in the used ring, it signals the need for an interrupt by simply writing to the irqfd.
    • KVM receives this signal and injects the corresponding virtual interrupt into the guest.
    • Result: The vhost-net thread can trigger a guest interrupt without ever needing to communicate with QEMU, eliminating another context switch.

With these optimizations, the data path stays almost entirely within the kernel:

Guest Kernel <-> KVM <-> Vhost Kernel Module <-> Host Kernel Network/Storage Stack

The only remaining overheads are the necessary VM exits for notifications and the data copies between guest RAM and host kernel buffers.

3. Observing Vhost in Action

Theory is great, but seeing is believing. Given your background, you'll appreciate how you can inspect a running system to verify that vhost is active and see the components interacting.

Hands on vhost-net: Do. Or do not. There is no try

This Red Hat blog post provides a hands-on guide to inspecting a vhost-net setup. It shows the exact commands you would use in a real-world scenario to confirm that offloading is working correctly.

Read the sections 'Inspecting the host', 'Generating traffic', and 'Analyzing the performance impact'. You don't need to run the commands yourself, but focus on the command output and the interpretation provided: How ps shows a vhost-[pid] kernel thread. How /proc/[qemu-pid]/fd shows a file descriptor for /dev/vhost-net. How strace on the QEMU process shows a dramatic reduction in ioctl and read/write syscalls when vhost is enabled.

Here’s a summary of the key verification steps from the article:

  1. Check for the kernel module:

    lsmod | grep vhost

    This confirms vhost_net, vhost, and vhost_iotlb are loaded.

  2. Identify the vhost kernel thread:

    ps -ef | grep vhost

    You will see a kernel thread named [vhost-<qemu_pid>] for each VM using vhost. This is the thread doing the data plane work.

  3. Inspect QEMU's connection to vhost:

    


# Find the QEMU process PID
pidof qemu-system-x86_64



# List its open file descriptors
ls -l /proc/<qemu_pid>/fd

    In the output, you will find a file descriptor pointing to /dev/vhost-net. This character device is the control plane interface that QEMU uses to configure the vhost-net kernel module (e.g., telling it where the virtqueues are in guest memory).

The performance analysis using iperf3 and strace powerfully demonstrates the impact. With vhost disabled, strace shows the QEMU process spending most of its time in read() and ioctl() syscalls. With vhost enabled, those calls virtually disappear, as QEMU is no longer in the data path.

Interview Scenario

An interviewer presents this situation:

"A customer is complaining about poor network throughput and high CPU usage on a host running several network-intensive VMs. They confirm they are using virtio-net drivers in the guests. From the host, you run top and notice that the qemu-system-x86_64 processes corresponding to these VMs have very high CPU utilization. What is your primary suspect for this performance issue, and what three commands would you run on the host to confirm your hypothesis?"

Click to see a model response

  1. Primary Suspect: The primary suspect is that vhost-net is disabled or misconfigured. The described symptoms—high qemu process CPU usage despite using virtio—are classic signs that the QEMU process is stuck handling the entire network data plane instead of offloading it to the kernel.

  2. Confirmation Commands:

    • First, I'd check if a vhost kernel thread exists for the VM:

      ps -ef | grep '[v]host'

      I'm looking for a [vhost-<pid>] thread corresponding to the problematic qemu process PID. If it's missing, vhost is not active for that VM. The [v] is a trick to prevent grep from matching its own process.

    • Second, I'd verify QEMU's configuration by inspecting its file descriptors:

      ls -l /proc/<qemu_pid>/fd | grep vhost-net

      If this command returns nothing, it confirms QEMU hasn't opened /dev/vhost-net, meaning it's not using the offload mechanism.

    • Third, to be thorough, I'd ensure the kernel module is loaded at all:

      lsmod | grep vhost_net

      While it's usually present on modern systems, its absence would be a definitive root cause.

    The absence of the vhost thread and the file descriptor in the QEMU process would confirm that the data plane is not being offloaded, explaining why the QEMU process itself is consuming so much CPU. The solution would be to correct the VM's configuration to enable vhost-net.

Conclusion

In this lesson, we bridged the gap between efficient virtio communication and true high-performance I/O by introducing the vhost protocol. By offloading the data plane, vhost removes the QEMU userspace process from the critical path of data transfer.

Key Takeaways:

  • The Bottleneck: The standard QEMU virtio backend is limited by context switches and syscalls as data moves between the host kernel and the QEMU userspace process.
  • The Solution: vhost-net and vhost-scsi are kernel modules that take over the virtio data plane, creating a dedicated kernel thread for each VM's I/O.
  • The Mechanism: QEMU handles the control plane (setup), while the vhost thread handles the data plane (virtqueue processing).
  • Key Enablers: ioeventfd and irqfd allow KVM to mediate notifications directly between the guest and the vhost kernel thread, effectively bypassing QEMU.
  • Verification: You can confirm vhost is active by looking for the vhost kernel thread with ps and checking for an open file descriptor to /dev/vhost-net in the QEMU process's /proc entry.

Preview of the Next Lesson:

We have now examined the architecture of virtio I/O from multiple performance angles. Next, we will shift our focus from performance to control and introspection. In the upcoming lesson, you will learn to use QEMU's monitor (HMP/QMP) to inspect machine state and memory mappings, giving you a powerful, interactive tool to debug and manage a running virtual machine.