PCIe Enumeration: Kernel Device Discovery & Configuration

1. The Goal: Discovering the Hierarchy

At power-on, the kernel's knowledge of the PCIe hierarchy is minimal. It only knows about the existence of the Root Complex and the bus directly attached to it, Bus 0. Everything else is a mystery.

The goal of enumeration is to perform a systematic scan to:

1. Discover every PCIe function on every device.
2. Assign each function a unique address, known as Bus, Device, Function (BDF).
3. Read configuration information to understand what each device is and what resources it needs.

To do this, the kernel reads from a special per-device data structure: the Configuration Space.

2. The Configuration Space: A Device's Identity Card

Every PCIe function has a dedicated memory region called the Configuration Space. This space contains all the information the kernel needs to identify and configure the device. While there is also I/O and memory space used for the device's main operation, the configuration space is purely for discovery and setup.

The first 64 bytes of the configuration space are the standardized PCI Header. This header contains the most critical information for enumeration.

As the video explained, the key fields for initial discovery are:

• Vendor ID (VID): A unique number assigned to each hardware vendor (e.g., 0x8086 for Intel).
• Device ID (DID): A number assigned by the vendor to identify a specific product.
• Class Code: Identifies the type of device (e.g., network controller, storage controller, display controller).
• Header Type: This is crucial for enumeration.
  • Type 0: Indicates a standard Endpoint.
  • Type 1: Indicates a PCI-to-PCI Bridge (including ports on a Switch or the Root Complex).

When the kernel scans a potential BDF address, a returned Vendor ID of 0xFFFF signifies that no device is present. Any other value indicates a device has been found. The combination of Vendor ID and Device ID uniquely identifies the hardware, which is how the kernel later matches it to a specific driver. The Header Type tells the kernel whether to continue scanning on the current bus or to start scanning a new bus behind a bridge.

3. The Enumeration Algorithm: A Depth-First Search

The kernel enumerates the PCIe fabric using a depth-first search algorithm. It systematically explores the tree of buses, starting from Bus 0.

Here is the simplified process:

1. Start with Bus 0: The scan begins on the root bus attached to the Root Complex.
2. Scan for Devices: The kernel iterates through all possible device numbers (0-31) on the current bus. For each device, it checks function 0.
3. Check for Device Presence: For each potential BDF (Bus X, Device Y, Function 0), the kernel attempts to read the Vendor ID from its configuration space.
   • If the result is 0xFFFF, there is no device at this location. The kernel moves to the next device number.
   • If a valid Vendor ID is returned, a device exists! The kernel reads the rest of its header.
4. Handle Multi-Function Devices: If the device has a multi-function header bit set, the kernel also scans functions 1-7 for that device.
5. Bridge or Endpoint? The kernel checks the Header Type:
   • If Header Type is 0 (Endpoint): The device is a "leaf" in the hierarchy. The kernel stores its information (pci_dev struct) and continues scanning the current bus.
   • If Header Type is 1 (Bridge): This device is a gateway to another bus segment. The kernel must now explore what's behind it.
     a. It configures the bridge's bus registers:
     • Primary Bus: Set to the current bus number.
     • Secondary Bus: Set to a new, unused bus number. This number is now assigned to the bus segment on the other side of the bridge.
       b. Recurse: The kernel calls the enumeration function recursively, passing the new secondary bus number as the target to scan.
       c. Update Subordinate Bus: After the recursive call returns (meaning everything behind the bridge has been scanned), the kernel updates the bridge's Subordinate Bus register to the highest bus number that was discovered behind it. This tells the system the full range of buses managed by this bridge.

This recursive process continues until all paths have been explored. The result is a complete map of the PCIe hierarchy, with every function on every device assigned a unique BDF.

4. Accessing the Configuration Space

How does the kernel actually perform the "read from configuration space" step? There are two primary mechanisms.

Legacy Mechanism (x86 I/O Ports)

On older PCI systems and for backward compatibility on x86, configuration space was accessed via two I/O ports:

• CONFIG_ADDRESS (0xCF8)
• CONFIG_DATA (0xCFC)

To read a register, the kernel would:

1. Write a 32-bit value to port 0xCF8 encoding the target BDF and register offset.
2. Read the 32-bit data value from port 0xCFC.

Modern Mechanism: Enhanced Configuration Access Mechanism (ECAM)

PCI Express introduced a more efficient memory-mapped approach called ECAM. The system firmware reserves a large, contiguous block of physical memory (up to 256 MiB) and maps the configuration space of every possible BDF into this region.

With ECAM, reading a configuration register becomes a simple memory read. The BDF and register offset are used to calculate an address within the ECAM memory region.

$\text{Address}=\text{ECAM\_Base}+(\text{Bus}\ll 20)|(\text{Device}\ll 15)|(\text{Function}\ll 12)|\text{Register}$

The kernel gets the ECAM_Base physical address from the platform firmware (ACPI or Device Tree). This allows the kernel to access any device's configuration space with a direct memory operation, which is much faster than using I/O ports.

5. Platform-Specific Implementations: ACPI vs. Device Tree

The enumeration algorithm is generic, but it needs a starting point. The kernel must first find the Root Complex and get its resources (like the ECAM base address). This information is provided by platform-specific firmware.

ACPI (Advanced Configuration and Power Interface)

Commonly used on x86 servers and desktops, ACPI provides the kernel with tables describing the hardware.

1. Firmware Stage: The BIOS/UEFI creates ACPI tables in memory, including the DSDT (Differentiated System Description Table). The DSDT contains a software description of the PCI Root Bridge.
2. Kernel Stage:
   • The kernel parses these tables to find the Root Bridge device object (often PNP0A03).
   • The acpi_pci_root driver binds to this object.
   • This driver reads the resources for the root bridge, including the bus number range and the ECAM configuration information from the ACPI MCFG table.
   • With this anchor point established, the kernel's generic PCI core can begin the enumeration process at Bus 0.

The LinkedIn post you read earlier (LINK) provides an excellent summary of this entire flow from Phase 1 to Phase 8.

Device Tree (DT)

In the embedded world, Device Tree is the standard.

1. Firmware Stage: The bootloader loads the Device Tree Blob (DTB) into memory and passes its address to the kernel.
2. Kernel Stage:
   • The kernel parses the DTB. It looks for a node that is compatible with a PCI host controller driver. This node describes the Root Complex.
   • The corresponding host controller driver initializes the hardware and gets the resource information (like memory ranges for ECAM) directly from its DT node.
   • Just like with ACPI, once the host bridge is initialized, the generic PCI core takes over and begins enumeration.
   • Dynamic Node Creation: A key feature in modern kernels used for embedded systems is CONFIG_PCI_DYNAMIC_OF_NODES. With this enabled, the base DTB only needs to describe the root complex. As the kernel enumerates and finds new PCI bridges and endpoints, it dynamically creates new device tree nodes for them in memory. This allows the rest of the kernel (e.g., drivers looking for their device) to interact with discovered PCI devices as if they were described in the original DTB.

6. Observing Enumeration and Debugging

The primary user-space tool to inspect the results of enumeration is lspci. It reads the information gathered by the kernel and displays the device hierarchy.

Interview Scenario

Interviewer: "You've just brought up a new embedded board with a custom FPGA connected via PCIe. The FPGA is supposed to appear as a simple memory device, but it's not visible in /proc/devices, and lspci doesn't list it. What are the first few steps you would take to debug this, and what are the most likely causes?"

Conclusion

In this lesson, we demystified the PCIe enumeration process. You've seen how the kernel transforms an unknown hardware topology into a fully mapped and identified device tree.

Key Takeaways:

• Enumeration is a depth-first search starting from Bus 0 to discover all PCIe functions.
• The kernel reads the Configuration Space Header of each potential device to find its Vendor/Device ID and, crucially, its Header Type.
• Header Type 1 (Bridges) triggers a recursive scan on a newly assigned bus number.
• Access to configuration space is done via the memory-mapped ECAM on modern systems.
• The entire process is anchored by platform firmware—ACPI on x86 or Device Tree on embedded systems—which provides the starting information about the Root Complex.

You now understand how the kernel finds a device. In the next lesson, "Write a PCIe driver skeleton that registers with the PCI subsystem and reads device configuration space," we will explore the next step: how a device driver finds and binds to the device that the kernel has just discovered for it.