PCIe Architecture: Layers, Root Complex, Switches, and Endpoints

1. The PCIe Topology: A Hierarchical System

Unlike its predecessor, the parallel PCI bus, which was a shared bus where multiple devices listened on the same set of wires, PCI Express (PCIe) uses a point-to-point topology. Each connection, called a Link, connects exactly two devices. To build a system with multiple devices, these links are arranged in a tree-like hierarchy.

The main components that form this hierarchy are:

• Root Complex (RC): The root of the tree. It connects the CPU and main memory to the PCIe fabric. You can think of it as the bridge between the processor's world and the PCIe world. The RC is typically integrated into the CPU or the motherboard's chipset.
• Endpoint (EP): The leaves of the tree. These are the actual devices you want to use, such as a graphics card, a network interface card (NIC), an NVMe SSD, or a custom device on an expansion card.
• Switch: The branches of the tree. A switch is used to expand the number of available PCIe ports. It has one "upstream" port connecting towards the Root Complex and multiple "downstream" ports connecting to Endpoints or even other switches.

To get a clear definition of these core components and see how they fit together, please watch the following segment.

This hierarchical structure is fundamental to how PCIe works. Any communication between two Endpoints must travel "upstream" towards a common point (a Switch or the Root Complex) and then "downstream" to the target.

The following article provides a clear textual description and a helpful diagram of this topology.

2. The Three Layers of PCIe Communication

Now that we understand the physical arrangement of components, let's explore how they communicate. The PCIe specification defines a three-layer protocol stack. Data that needs to be sent is packaged at the top layer and passed down, with each layer adding its own information, until it is sent as electrical signals over the physical link. The receiving device reverses the process.

This layered approach is a classic engineering solution that separates concerns, which should be familiar from your work with complex embedded systems and network stacks.

Here is a breakdown of each layer's primary responsibility:

1. Transaction Layer: This is the highest layer and the one that software most directly interacts with. Its job is to create and interpret Transaction Layer Packets (TLPs). These packets are requests or completions for specific actions, such as:

   • Memory Read/Write
   • I/O Read/Write
   • Configuration Read/Write (used for device discovery, which we'll cover next lesson)
   • Messages (for events like interrupts and errors)
     When your driver initiates a DMA transfer, the hardware is ultimately creating and processing TLPs at this layer.

2. Data Link Layer: This middle layer acts as a reliable transport service for the Transaction Layer. Its key responsibilities are to ensure data integrity and manage flow control. It takes TLPs from the layer above and adds:

   • A sequence number to track packets.
   • A Link CRC (LCRC) for error detection.
     This layer is responsible for acknowledging (ACK) correctly received packets and requesting re-transmission (NAK) for corrupted ones.

3. Physical Layer: This is the lowest layer, responsible for the actual transmission of data over the physical link. It handles:

   • Serializing the parallel data from the Data Link Layer into a serial bitstream.
   • Encoding the data to ensure DC balance and provide a guaranteed number of signal transitions for clock recovery (e.g., 8b/10b or 128b/130b encoding).
   • The electrical signaling across one or more Lanes.
   • Link initialization and training (negotiating link width and speed).

To see how these layers work in concert, the following reading provides an excellent summary.

3. Interview Scenario: Deconstructing a System

To solidify these concepts, let's analyze a typical scenario you might encounter in a high-level interview.

Scenario:
You are examining the architecture of a modern server. It contains a CPU, two NICs for redundant networking, and four NVMe SSDs configured in a RAID array for high-speed storage. All peripherals are PCIe-based.

Questions:

1. Describe a likely PCIe topology for this system. Identify the Root Complex, the Endpoints, and where a Switch would be necessary.
2. Trace the high-level journey of a single "Memory Write" TLP from the CPU to one of the NVMe SSDs. Describe what happens at each of the three protocol layers at both the source (Root Complex) and destination (NVMe Endpoint).

Conclusion

In this lesson, we established the foundational model of the PCIe subsystem. You now have a mental map of how PCIe devices are physically interconnected and how they communicate through a structured, layered protocol.

Key Takeaways:

• PCIe Topology: It is a hierarchical, point-to-point system, not a shared bus. The core components are the Root Complex (CPU bridge), Endpoints (devices), and Switches (port expanders).
• Layered Architecture: Communication is managed by three distinct layers:
  • The Transaction Layer creates packets for memory, I/O, and configuration operations.
  • The Data Link Layer ensures reliable, error-checked delivery of those packets.
  • The Physical Layer handles the electrical signaling and data encoding on the wire.

This understanding is critical. As we move forward, every concept—from device discovery to interrupt handling—builds upon this model of a hierarchical topology and a layered protocol.

In our next lesson, we will address a fundamental question: When a computer boots up, how does the kernel know what PCIe devices are present? We will explore PCIe enumeration and how the kernel discovers and configures devices during boot.