Distributed-Memory Architectures and Interconnection Networks

From Shared to Distributed Memory

In a shared-memory system, communication between cores is implicit. A write to a memory location by one core is made visible to others through hardware coherence mechanisms.

In a distributed-memory system, communication is explicit. Each processor, or node, has its own private memory. A processor cannot directly access another processor's memory. To share data, one processor must explicitly package the data into a message and send it to another processor, which must explicitly receive it.

The image below provides a clear visual distinction between these two models.

To see a narrated explanation of this distinction, let's watch a brief segment of the following video.

As the video explains, accessing remote data in a distributed system is not a single instruction but a coordinated procedure of sending and receiving messages. This is the foundation of the message-passing programming paradigm.

The Message-Passing Model

Programs for distributed-memory systems are typically written using the message-passing model. The de facto standard library for this is the Message Passing Interface (MPI). While we will dive deep into MPI programming in a later module, understanding its core concepts is essential for grasping the architecture.

In the MPI model:

• A program is launched as a set of independent processes.
• Each process is assigned a unique integer ID called its rank.
• All processes belong to a communication group called a communicator. The default, MPI_COMM_WORLD, includes all processes.
• Processes communicate by calling functions like MPI_Send and MPI_Recv.

As you read, note the two main advantages of the distributed-memory model:

1. No data corruption risk: Since memory is private, one process cannot accidentally overwrite another's data.
2. Memory scalability: The total system memory scales with the number of processors, unlike a shared-memory system, which is limited by a single physical memory unit.

The Role of the Interconnection Network

The collection of nodes in a distributed-memory system is often called a cluster. The performance of such a system is heavily dependent on the communication fabric that connects the nodes: the interconnection network. This network consists of:

• Endpoints: The processors/nodes themselves.
• Links: The physical wires or optical fibers that carry data.
• Switches: Devices that route messages from an incoming link to the correct outgoing link, directing them toward their destination.

The way these components are arranged is called the network topology. The topology determines the performance, cost, and scalability of the entire parallel machine.

Key Network Performance Metrics

As outlined in the notes, we evaluate network topologies using a few key metrics:

• Diameter: The longest shortest path between any two nodes in the network. This represents the worst-case latency for sending a message, assuming no network congestion. A smaller diameter is better.
• Bisection Bandwidth: The minimum number of links you must cut to divide the network into two equal halves. This measures the network's communication bottleneck and is a crucial indicator of its ability to handle global communication patterns where many nodes communicate simultaneously. A larger bisection bandwidth is better.

• Degree: The number of links connected to a single node or switch. This is a primary indicator of the network's hardware cost and complexity.

An ideal network has a low diameter, high bisection bandwidth, and a low degree. In practice, there are always trade-offs.

Common Network Topologies

Let's explore some common topologies and their trade-offs. These are categorized as either direct networks, where processors are part of the network fabric, or indirect networks, where processors connect to a separate network of switches.

Here is a summary of the key topologies you just read about:

Your engineering background gives you a good intuition for grid-based problems. A 2D Mesh topology is a natural fit for simulating a 2D physical space, as each node can directly communicate with its north, south, east, and west neighbors, minimizing communication distances for local interactions.

The Fat-Tree is particularly important. Its design ensures that bandwidth does not decrease as you move up the hierarchy toward the "root," avoiding the bottleneck that a simple tree structure would create. This makes it excellent for all-purpose clusters where communication patterns are not predictable.

Conclusion

In this lesson, we have moved from the tightly-coupled world of shared memory to the scalable but more complex realm of distributed memory.

Key Takeaways:

• Distributed-memory architectures consist of independent nodes, each with its own private memory. Communication is explicit, achieved by sending and receiving messages.
• The Message Passing Interface (MPI) is the standard programming model for these systems.
• The interconnection network is the physical fabric connecting the nodes, and its topology is critical to the system's performance and scalability.
• Network performance is evaluated using metrics like diameter (latency) and bisection bandwidth (throughput).
• Common topologies like Ring, Mesh, and Torus are direct networks with different cost-performance trade-offs.
• The Fat-Tree is a highly scalable indirect network topology used in many modern high-performance clusters because it avoids communication bottlenecks.

Preview of the Next Lesson

We have now examined two of the main parallel architectures: shared-memory and distributed-memory systems. In our next lesson, we will explore a third, highly influential architecture: the Graphics Processing Unit (GPU). We will describe its unique structure, including streaming multiprocessors, warps, and its specialized memory hierarchy. You will see how GPUs represent a massively parallel architecture that combines elements of both shared and distributed models to achieve incredible computational throughput for specific types of problems.