Shared vs. Distributed vs. GPU: System Trade-offs

1. Architectural and Memory Model Comparison

The most fundamental difference between these systems lies in how their processors access memory. This architectural choice has profound implications for programming, communication, and scalability.

Shared vs. Distributed Memory

Let's start with a quick visual recap of the two primary CPU-based models.

• Shared-Memory Systems:

  • Architecture: All processor cores share a single global address space.
  • Communication: Implicit and low-latency. One core writes to a memory location, and another core can read from it (managed by cache coherence protocols).
  • Limitation: Scalability is limited. As you add more cores, the shared bus or interconnect becomes a bottleneck, and the overhead of keeping caches coherent increases. It's difficult to build a single system with thousands of cores sharing memory this way.

• Distributed-Memory Systems:

  • Architecture: Each processor (or node) has its own private memory. There is no global address space.
  • Communication: Explicit and higher-latency. Processors communicate by sending and receiving messages over a network.
  • Advantage: Highly scalable. You can connect thousands of nodes to build a massive supercomputer, as memory bandwidth and access are not a central bottleneck.

The GPU Model: A Specialized Hybrid

A GPU introduces a third, more complex model. It acts as a co-processor or accelerator attached to a CPU (the host).

• Architecture: The GPU (device) has its own, separate memory space (its onboard VRAM, called "global memory"). This memory is physically distinct from the CPU's main memory (RAM).
• Communication:
  • Host-Device: Explicit and a major performance bottleneck. Data must be copied from CPU RAM to GPU VRAM before computation and results copied back after. This is typically done over a PCIe bus.
  • On-Device: The GPU has its own sophisticated memory hierarchy, which you explored in the last lesson. Communication between threads on the GPU is managed through this hierarchy, with extremely fast on-chip shared memory being crucial for performance.

2. Programming Models and Complexity

The underlying architecture dictates the programming model used to harness its power. We'll associate these architectures with their most common programming APIs: OpenMP (shared), MPI (distributed), and CUDA (GPU).

As you've just read, there's a clear spectrum of complexity:

• Shared Memory (OpenMP): Generally considered the easiest to use. You can often parallelize existing C++ code incrementally by adding compiler directives (#pragma omp ...) to loops. Communication is implicit—you just use shared variables. The main challenge is correctly using synchronization primitives (like critical, atomic, barrier) to prevent race conditions.
• Distributed Memory (MPI): The most complex model. You are responsible for everything: partitioning the data across processes, explicitly sending and receiving data with functions like MPI_Send and MPI_Recv, and ensuring processes are synchronized. The potential for deadlocks and the cognitive overhead of managing data flow is high.
• GPU (CUDA): Has a steep learning curve. It requires you to think in terms of massive parallelism and write separate functions called kernels that execute on the GPU. The biggest challenge is managing the complex memory hierarchy—moving data between the CPU and GPU, and using on-chip shared memory effectively to minimize slow global memory access. This often requires significant code restructuring compared to a standard CPU algorithm.

3. Performance, Scalability, and Application Suitability

This is the heart of the trade-off. Each system is designed to excel at a different kind of task. A key distinction is between being latency-optimized (like a CPU) and throughput-optimized (like a GPU).

A CPU core is a "jack of all trades"—fast and versatile, designed to execute a single thread of complex, branching instructions with minimum delay (low latency). A GPU is a specialist, packed with thousands of simple cores that execute simple instructions in parallel to process massive datasets (high throughput).

Let's look at some quantitative data. Given your background in engineering, seeing the real-world performance impact will be insightful.

The results from that paper are striking:

• In the finite-difference test, the GPU was 20 to 100 times more efficient than the hybrid CPU cluster model.
• For the Reverse Time Migration task, a single GPU on a desktop computer completed the job in 1/4 of the time it took a 10-node CPU cluster, at a fraction of the hardware cost and energy consumption.

This brings us to a summary of which problems fit each architecture best.

4. The Rise of Hybrid Models

As you may have noticed, these architectures are not mutually exclusive. The most powerful supercomputers today are heterogeneous, combining the strengths of all three models. A typical high-performance computing (HPC) node consists of:

• Multiple CPUs with several cores each (a shared-memory system).
• One or more GPUs attached as accelerators.
• Many of these nodes are connected via a high-speed network (a distributed-memory system).

Programming these systems often involves a hybrid approach:

• MPI + OpenMP: MPI is used for communication between nodes, and OpenMP is used to exploit all the CPU cores within a single node.
• MPI + CUDA: MPI is used to distribute work across multiple nodes, each of which uses its GPU(s) to perform the heavy computation.

This allows developers to map the different levels of parallelism in their application to the most appropriate hardware level.

Conclusion

We've now completed our survey of the foundational parallel architectures. You should have a clear understanding of the critical trade-offs involved when choosing a parallel system.

Key Takeaways:

• No single "best" architecture exists. The optimal choice is driven by the problem's characteristics: its data size, communication patterns, and type of parallelism.
• Shared-memory systems (multi-core CPUs) are easy to program and are great for general-purpose, latency-sensitive, or task-based parallelism on a single machine.
• Distributed-memory systems (clusters) offer near-limitless scalability for problems too large for a single machine but come with high programming complexity due to explicit message passing.
• GPU-based systems provide unmatched computational throughput for massively data-parallel problems but require specialized programming and careful management of data movement.
• Modern scalable systems are often hybrid, combining these models to maximize performance.

Preview of the Next Lesson

Having established the "what" and "why" of parallel architectures, we will now transition to the "how." In the next module, "Principles of Parallel Algorithm Analysis," we will begin by learning how to formally analyze code to find opportunities and constraints for parallelization. Our first lesson will focus on identifying data dependencies (RAW, WAR, WAW), the fundamental constraints that determine whether a piece of code can be safely executed in parallel.