Chiplets: Design, Cost, Yield, and Specialization

Introduction

Welcome back. In our previous lesson, we examined the architecture of specialized accelerators like Google's TPU, which use systolic arrays to achieve incredible efficiency for matrix multiplication. We established that this performance comes at the cost of flexibility, as these are monolithic Application-Specific Integrated Circuits (ASICs) designed for a narrow purpose. This raises a critical question for system designers: is there a middle ground between the flexibility of a general-purpose GPU and the rigid specialization of an ASIC?

This lesson addresses that question by focusing on the learning outcome: Explain the chiplet-based approach to processor design and its implications for cost, yield, and specialization. We will explore how decomposing a large, single chip into a collection of smaller, interconnected "chiplets" is revolutionizing processor design. We'll analyze why this approach is becoming necessary and how it directly impacts manufacturing cost, production yield, and the ability to create highly specialized, heterogeneous systems for workloads like AI.

The Limits of Monolithic Scaling

For decades, the semiconductor industry has followed Moore's Law by shrinking transistors to pack more logic onto a single piece of silicon, a so-called monolithic die. This created ever-more-powerful System-on-a-Chip (SoC) designs. However, this approach is hitting fundamental economic and physical walls.

To understand why, let's start with a video that explains the challenges facing traditional monolithic chip design.

As the video explains, the core problem is one of cost and probability. Manufacturing a large, complex chip on a silicon wafer is an imperfect process. Microscopic defects are inevitable.

• On a large monolithic die, a single unrepairable defect can render the entire multi-billion transistor chip useless.
• The larger the die, the higher the probability it will contain such a defect. This leads to poor yield—the percentage of usable chips per wafer.

This relationship between die size and yield is the primary economic driver behind the chiplet paradigm.

The Chiplet Approach: Divide and Conquer

The chiplet-based approach fundamentally rethinks chip design. Instead of building one massive, monolithic SoC, the design is disaggregated into several smaller, independent dies called chiplets. Each chiplet might contain a specific function—CPU cores, a GPU, I/O controllers, or memory interfaces. These chiplets are then integrated and connected on a common substrate or package.

The following video segments continue this discussion, defining the chiplet concept and contrasting it with the historical precedent of Multi-Chip Modules (MCMs).

Now let's systematically explore the three major implications of this design philosophy.

Implication 1: Cost and Yield

As we've established, the most immediate benefit of chiplets is economic. By breaking a large design into smaller pieces, manufacturing yield improves dramatically.

Imagine a wafer has a certain number of random defects.

• If your wafer fits 100 large monolithic dies, and 50 have defects, your yield is only 50%.
• If your wafer fits 1,200 small chiplets, and 50 have defects, you still have 1,150 usable dies. Your yield is over 95%.

This effect is so pronounced that it can drastically lower manufacturing costs, even after accounting for the added complexity of packaging multiple dies together.

Academic analysis confirms this. A detailed cost model shows that the recurring engineering cost of a chiplet-based system can be nearly half that of an equivalent monolithic one.

Implication 2: Specialization and Flexibility

Beyond cost, chiplets unlock a new level of design flexibility and specialization, which is highly relevant to your goal of designing complex AI systems.

A. Mix-and-Match Process Nodes

As Sam Naffziger explained, not all components on a chip benefit equally from the expensive, cutting-edge fabrication processes.

• Core Logic (CPUs/GPUs): Scales well with new nodes, gaining performance and power efficiency.
• Analog I/O (e.g., PCIe, USB controllers): Scales very poorly. Moving an I/O controller from a 14nm process to a 5nm process provides little benefit but costs much more.
• Memory (SRAM caches): Scales, but not as well as logic.

Chiplets allow designers to use the right process for the right job. The compute-intensive logic can be placed on chiplets made with an advanced 5nm node, while the I/O functions can be placed on a separate I/O die made with an older, cheaper, and more mature 14nm or 6nm node. This optimizes both cost and performance.

B. Modularity and Reusability

Once you have designed a specific chiplet, for example, an 8-core CPU chiplet, you can reuse it across a wide range of products.

• A high-end server CPU might use eight of these chiplets for 64 cores.
• A mainstream desktop CPU might use one or two of these chiplets.
• A workstation CPU might use four.

This modularity allows companies to address different market segments with a much smaller number of unique components, amortizing the immense non-recurring engineering (NRE) costs of chip design across a larger volume of products.

C. Domain-Specific Architectures

Perhaps most excitingly for AI, chiplets enable the creation of highly specialized processors by combining different types of chiplets. Instead of being limited to what can be integrated onto one monolithic die, designers can build systems like Lego blocks. For example, one could design a processor for deep learning training by combining:

• Several GPU compute chiplets (on a 5nm node).
• A dedicated memory-system chiplet with a massive on-package cache and multiple HBM controllers (on a 7nm node).
• An I/O chiplet for high-speed networking (on a 12nm node).

This composability allows for creating processors that are highly tailored to a specific domain, like deep learning, without the risk and inflexibility of a fully monolithic ASIC.

The Catch: Performance Overheads and Interconnects

Chiplets are not a free lunch. The primary challenge is communication. On a monolithic chip, wires are incredibly short, fast, and dense. Communicating between two separate pieces of silicon (chiplets) is inherently slower, consumes more power, and is limited by the number of physical connections you can make.

This has two major consequences:

1. Interconnect Bottlenecks: The fabric connecting the chiplets (like AMD's Infinity Fabric) becomes a critical performance component. Its latency and bandwidth can limit the overall system performance. This is especially true for GPUs, which have immense internal bandwidth requirements that are difficult to satisfy with off-chiplet links.
2. On-Package NUMA: Chiplet-based systems create a Non-Uniform Memory Access (NUMA) environment within a single processor package. A compute unit accessing memory on its local chiplet will see much lower latency and higher bandwidth than when accessing memory attached to a remote chiplet. This adds a layer of complexity that software and schedulers must manage.

Conclusion

In this lesson, we have dissected the chiplet-based design paradigm, understanding it as a strategic response to the end of easy monolithic scaling. You now have a framework for analyzing the profound trade-offs this approach presents to a system architect.

Key Takeaways:

• Chiplet Motivation: The chiplet approach is driven by the high cost and poor yield of large, monolithic dies on advanced fabrication nodes.
• Implication for Cost & Yield: Breaking a large chip into smaller chiplets dramatically increases manufacturing yield, which can cut the final chip cost by almost half.
• Implication for Specialization: Chiplets enable a "mix-and-match" strategy. Designers can use the optimal process node for each function (e.g., 5nm for logic, 14nm for I/O) and create domain-specific processors by combining different types of chiplets.
• The Trade-off: The primary challenge is the interconnect. Communication between chiplets is slower and more power-hungry than on-die communication, introducing performance overheads and creating an on-package NUMA architecture that software must account for.

The chiplet approach moves the design challenge from the transistor level to the system integration level. It's less about cramming everything onto one die and more about intelligently partitioning a system and connecting the pieces efficiently.

Preview of the Next Lesson:

We have identified the inter-chiplet interconnect as the most critical and challenging aspect of this new design paradigm. For a true "Lego-like" ecosystem to flourish, where a designer can combine chiplets from different vendors, there must be a common standard for communication. Our next lesson will dive into exactly that: we will describe emerging interconnect standards (e.g., CXL, UCIe) and their potential to create more heterogeneous and composable systems.