Emerging Interconnect Standards for Heterogeneous and Composable Systems

Introduction

Welcome to the final lesson of our course on emerging hardware for AI. In our last session, we explored the chiplet-based design paradigm, concluding that while it solves critical issues of cost and yield, it introduces a new challenge: the interconnect. Proprietary on-package interconnects limit the dream of a true "Lego-like" ecosystem where components from different vendors can be seamlessly integrated. This bottleneck highlights the need for open standards.

This lesson directly addresses that need, focusing on the learning outcome: Describe emerging interconnect standards (e.g., CXL, UCIe) and their potential to create more heterogeneous and composable systems. We will investigate two pivotal standards that are shaping the future of system design.

• Compute Express Link (CXL), which redefines the relationship between processors, memory, and accelerators at the system level.
• Universal Chiplet Interconnect Express (UCIe), which aims to standardize the physical links between chiplets inside a single package.

By the end of this lesson, you will understand how these two standards, both individually and together, are unlocking a new era of composable and heterogeneous computing tailored for the demands of AI.

Part 1: Compute Express Link (CXL) – Breaking the Memory Wall

Modern CPUs have seen a rapid increase in core counts, but the memory bandwidth and capacity available to each core have not kept pace. This growing disparity is often called the "memory wall." Adding more memory channels directly to a CPU is difficult and expensive due to physical pin count and board routing complexity. CXL was created to solve this problem.

To begin, let's understand the industry megatrends that created the need for a new type of interconnect.

CXL is an open industry standard that builds upon the physical and electrical interface of PCI Express (PCIe), which is ubiquitous in modern computers. However, it adds new protocols on top of PCIe to enable low-latency, coherent memory sharing between a host CPU and attached devices.

CXL vs. PCIe: A Fundamental Shift

While PCIe is excellent for bulk data transfers (like reading a file from an SSD), it operates on a non-coherent, memory-mapped I/O (MMIO) model. A CPU accessing memory on a PCIe device cannot cache that data, leading to high latency for fine-grained access. CXL fundamentally changes this by introducing coherency.

The next segment explains how CXL augments the PCIe infrastructure.

The CXL Protocol Suite

As the video explained, CXL's power comes from a set of three multiplexed protocols that run over the PCIe physical layer:

1. CXL.io: This is the base protocol and is functionally equivalent to PCIe. It handles device discovery, configuration, and non-coherent bulk data transfers. It is mandatory for all CXL devices.
2. CXL.cache: This optional protocol allows a CXL device (like an AI accelerator) to request and cache data from the host CPU's memory with coherency maintained by the host. This is crucial for accelerators that need low-latency access to shared data structures.
3. CXL.mem: This optional protocol allows the host CPU to access memory attached to a CXL device as part of its own coherent memory space. This is the key to memory capacity expansion.

This flexible combination of protocols allows for different types of devices tailored to specific needs.

How CXL Enables Composable Systems

CXL's architecture opens the door to fundamentally new ways of building servers and datacenters.

1. Memory Capacity and Bandwidth Expansion:
CXL Type-3 devices act as memory expansion modules. From the operating system's perspective, this CXL-attached memory simply appears as a new NUMA (Non-Uniform Memory Access) node. This allows servers to break free from the constraints of motherboard DIMM slots and add terabytes of memory via the PCIe bus, effectively creating a new tier of "far memory" that is slower than local DRAM but vastly faster than SSDs.

2. Disaggregation and Pooling (CXL 2.0 and beyond):
Later versions of the CXL standard introduce CXL switches. These switches allow for much more complex topologies.

• A pool of memory can be shared among multiple CPUs.
• A single accelerator can be accessed by multiple CPUs.
• Resources can be dynamically composed and assigned to workloads as needed.

This leads to the concept of disaggregated infrastructure, where compute, memory, and accelerators are no longer tightly bound in a single server chassis but exist as pools of resources in a rack, connected by a CXL fabric.

Part 2: Universal Chiplet Interconnect Express (UCIe) – The "USB for Chiplets"

While CXL addresses system-level and rack-level connectivity, UCIe focuses on the challenge we identified in the last lesson: standardizing the on-package, die-to-die interconnect. For a true open chiplet ecosystem to thrive, there needs to be a common interface for connecting chiplets from different vendors. UCIe aims to be that interface.

The motivation for UCIe directly follows from the reasons for adopting chiplets in the first place.

UCIe is an open specification that defines everything needed for two chiplets to talk to each other inside a package, from the electrical characteristics and physical bump layout to the protocol stack that ensures reliable communication.

The UCIe Architecture

UCIe provides a standardized physical layer (PHY) and a die-to-die adapter layer. It is designed to be a highly efficient transport for other, higher-level protocols.

Key takeaways from the UCIe architecture:

• Extremely High Performance: As a short-reach on-package interconnect, UCIe achieves bandwidth densities and power efficiencies that are orders of magnitude better than off-package interconnects like PCIe. We're talking terabytes per second of bandwidth per square millimeter, at picojoules per bit of energy.
• Protocol Agnostic: The UCIe stack is designed to carry other protocols. It has native support for tunneling PCIe and CXL, which is a critical point of synergy we will discuss shortly.
• Supports Multiple Packaging Technologies: It defines standards for both cost-effective 2D "standard" packages and high-performance 2.5D/3D "advanced" packages with silicon interposers.

How UCIe Enables Heterogeneous Systems

UCIe is the foundational layer for a modular, open chiplet ecosystem.

1. A "Mix-and-Match" Marketplace:
By providing a plug-and-play standard, UCIe enables the vision of building a complex System-in-Package (SiP) by sourcing chiplets from different vendors. An AI system designer could combine:

• A CPU chiplet from one company.
• A GPU compute chiplet optimized for AI from another.
• A specialized memory controller chiplet from a third.

This allows for unprecedented specialization and competition, accelerating innovation at the package level.

2. Extending Reach with Retimers:
A fascinating usage model for UCIe is to connect an electrical chiplet (like a CPU or GPU) to an optical chiplet on the same package. The optical chiplet then acts as a "retimer," converting the electrical UCIe signals into light and sending them over optical fiber. This allows the immense bandwidth of on-package interconnects to be extended across an entire datacenter rack with very low latency, enabling massive, composable systems.

Synergy: CXL and UCIe as a Foundation for Future Systems

It's crucial to understand that CXL and UCIe are not competitors; they are complementary technologies operating at different levels of the system stack.

• UCIe is the physical, on-package interconnect. Think of it as the standardized "Ethernet port and cable" for chiplets. It provides the raw, high-speed pipe.
• CXL is the logical, system-level protocol. Think of it as the "TCP/IP" that runs over the wire. It provides the language for memory coherency and resource sharing.

A future, highly advanced AI accelerator could be a System-in-Package (SiP) composed of multiple compute and memory chiplets. These chiplets would communicate with each other on the package using UCIe as the physical transport, and the protocol running over that transport would be CXL.cache and CXL.mem to maintain a single, coherent memory space across all chiplets. This entire SiP would then connect to the host CPU and the rest of the datacenter using a standard, off-package CXL link.

Conclusion

In this lesson, we have explored two of the most important emerging standards that are paving the way for the future of heterogeneous and composable systems.

Key Takeaways:

• CXL solves the memory bottleneck at the system level by providing a coherent protocol over the PCIe physical layer. It enables memory expansion, tiering, and, with switching, the disaggregation and pooling of resources like memory and accelerators.
• UCIe solves the proprietary interconnect problem at the package level. It provides an open, high-performance, low-power standard for connecting chiplets, enabling a true "mix-and-match" ecosystem.
• Together, they are synergistic. CXL provides the "what" (coherent memory protocol), while UCIe provides the "how" (a physical standard for die-to-die transport). They allow system innovation to move from the monolithic die to the package and the rack.

This lesson marks the end of our course. We have journeyed from the core architecture of a single GPU, through its memory hierarchy and programming models, into the complexities of parallelizing ML workloads and scaling them across massive, distributed systems. We've examined optimization techniques for training and inference, and finally, looked ahead to the emerging architectures that will power the next generation of AI. You now possess a comprehensive, foundational understanding that bridges the gap from high-level AI models to the underlying hardware systems that execute them.