Accessing PCIe Device BARs for MMIO

1. From Configuration Space to Operational Memory: BARs and MMIO

In the last lesson, we visualized the PCI configuration space using the following diagram. We focused on the Vendor and Device IDs. Now, let's turn our attention to the registers labeled Base Address Registers.

A BAR is more than just a register; it's a device's formal request for a block of memory in the host system's physical address space. During boot, the BIOS or kernel performs the following negotiation:

1. The Device Request: The device uses its BAR registers to declare its needs. For example, it might say, "I need a 4 KB region of memory-mapped I/O space." The hardware indicates this by allowing the kernel to write all 1s to the BAR and then read back the size mask.
2. The Host Allocation: The kernel finds an available, appropriately sized, and aligned chunk of physical memory in the system's address map.
3. The Assignment: The kernel writes the starting physical address of that allocated chunk back into the device's BAR register.

Once this is complete, a window is opened. Any CPU memory access to that specific range of physical addresses is no longer directed to system RAM; instead, the root complex routes it over the PCIe bus to the device. This mechanism is called Memory-Mapped I/O (MMIO). From the driver's perspective, the device's control registers and internal memory now appear as if they are part of the system's main memory.

To deepen your understanding of this architectural concept, the following video provides an excellent overview.

An important takeaway is that BARs are incredibly flexible. While they can map a device's onboard RAM (like a video card's framebuffer), they are more commonly used to expose a set of control and status registers. Writing a specific value to a particular address in a BAR might trigger an action on the device, while reading from another might return the device's current status.

2. The Driver's Workflow for Accessing BARs

Now, let's translate this architectural concept into a sequence of operations within our driver's probe function. To access a BAR, you must perform the following steps in order:

1. Enable the device: Wake up the device and enable its memory/IO resources.
2. Request the memory regions: Formally claim the memory regions covered by the BARs to prevent conflicts with other drivers.
3. Map the physical address to a virtual address: The kernel operates in a virtual address space. You cannot directly use the physical address from the BAR. You must map it to get a kernel virtual address (a void __iomem * pointer) that your driver can use.
4. Access the device: Use special kernel functions (ioread*/iowrite*) to read from and write to the mapped virtual address.

Modern Resource Management: Managed APIs

In kernel development, correctly managing resources (memory, I/O ports, etc.) is paramount. Forgetting to release a resource in your remove function can lead to leaks and system instability. To address this, the kernel provides a "managed" API, often prefixed with devm_ or pcim_.

These functions tie the resource's lifetime to the device itself. When you allocate a resource with a managed function, the kernel automatically frees it when your driver unbinds from the device. This simplifies your remove function significantly and makes your code more robust. It is the modern, preferred approach.

3. Practical Implementation: Mapping and Accessing a BAR

Let's apply these steps to our driver. We will extend the probe function from the previous lesson. The following video demonstrates the entire process. While the narrator begins by recommending managed functions, he ends up using a mix of managed and unmanaged APIs. We will use the video as a visual guide but will focus on implementing the fully managed, modern approach in our code.

Now, let's write the code for our probe function using the modern, fully managed API. This consolidates several steps shown in the video into safer, more concise calls.

Assume BAR_NUMBER is the BAR we want to access (e.g., 0).

#include <linux/pci.h>
#include <linux/io.h> // For ioread/iowrite

// A pointer for our mapped BAR space
static void __iomem *hwmem;

static int my_pci_probe(struct pci_dev *dev, const struct pci_device_id *id)
{
    int bar_num = 0; // Example: using BAR 0
    int ret;
    u32 reg_val;
    u16 vendor_id, device_id;

    pr_info("My PCI Driver: probe function called for device %04x:%04x\n", dev->vendor, dev->device);
    
    // 1. Enable the PCI device
    ret = pcim_enable_device(dev);
    if (ret) {
        pr_err("Failed to enable PCI device\n");
        return ret;
    }

    // 2. Request ownership of the BAR regions
    // pcim_iomap_regions automatically calls pci_request_regions
    ret = pcim_iomap_regions(dev, 1 << bar_num, "my_pci_driver");
    if (ret) {
        pr_err("Failed to request/map PCI regions\n");
        // pcim_enable_device cleanup is automatic
        return ret;
    }

    // 3. Get the mapped virtual address
    // The pcim_iomap_table is an array of pointers for each BAR
    hwmem = pcim_iomap_table(dev)[bar_num];
    if (!hwmem) {
        pr_err("Failed to get virtual address for BAR %d\n", bar_num);
        // pcim_iomap_regions cleanup is automatic
        return -EIO;
    }
    
    pr_info("BAR %d mapped at virtual address %p\n", bar_num, hwmem);

    // 4. Access the device
    // Example: Read a 32-bit register at offset 0x00 from the BAR
    reg_val = ioread32(hwmem + 0x00);
    pr_info("Read value 0x%08x from BAR %d at offset 0x00\n", reg_val, bar_num);

    // Example: Write an 8-bit value to a register at offset 0x04
    iowrite8(0xAB, hwmem + 0x04);
    pr_info("Wrote 0xAB to BAR %d at offset 0x04\n", bar_num);

    pci_set_master(dev); // Enable Bus Mastering for DMA, if needed later

    return 0;
}

In the corresponding remove function, we no longer need to manually unmap and release resources. The pcim_* wrappers handle this automatically when the driver is unbound.

static void my_pci_remove(struct pci_dev *dev)
{
    pr_info("My PCI Driver: remove function called for device %04x:%04x\n", dev->vendor, dev->device);
    // All resources allocated with pcim_* are freed automatically.
    // hwmem is also automatically unmapped.
}

Why use ioread/iowrite?

You must never directly dereference an __iomem pointer (e.g., *hwmem = 0xAB;). You must use the ioread*/iowrite* family of functions.

These functions are essential for two main reasons:

1. Preventing Compiler Optimizations: The compiler doesn't know that this memory location is special. It might reorder, combine, or even eliminate reads/writes, which would break the device's expected communication protocol. These functions ensure the access happens exactly as you wrote it.
2. Portability and Memory Barriers: On some architectures, I/O memory requires special CPU instructions or memory barriers to ensure correct ordering with other operations. These functions abstract away those platform-specific details.

This article provides a good textual reference for this concept.

Interview Scenario

Interviewer: "You've written a driver that maps a device's BAR0. When you read from a status register at offset 0x10, you consistently get back 0xFFFFFFFF. You've confirmed with lspci -v that BAR0 is a valid 64-bit memory region. What are the most likely causes of this issue in your driver code?"

Conclusion

Today you have taken a major step in PCI driver development. You've learned how to bridge the gap between the CPU and a device's operational registers, enabling true hardware control.

Key Takeaways:

• BARs define memory windows that devices use to expose their internal registers and memory to the host system.
• MMIO is the mechanism that allows the CPU to access these device windows as if they were regular RAM.
• The driver workflow is: enable device -> request regions -> map BAR -> access with ioread/iowrite.
• Using managed APIs (pcim_*, devm_*) is the modern best practice for robust resource management in the kernel, simplifying code and preventing resource leaks.
• All MMIO access must use special functions like ioread32 and iowrite32 to ensure correctness and portability.

We can now read from and write to our device, giving it commands and checking its status. But how does the device notify us when something important happens, like data becoming available or an error occurring? This is the role of interrupts. In our next lesson, we will explore how to handle hardware interrupts, focusing on the modern MSI and MSI-X interrupt mechanisms.