Message Passing & SPMD

1. From Shared to Distributed Memory

In the shared-memory model, parallelism is achieved using threads that share a single address space. In the distributed-memory model, we work with processes, each with its own private, isolated memory.

• Shared Memory: Imagine a team of engineers working on a single large whiteboard. Everyone can see, add, and erase information simultaneously. This is fast and direct, but they must be careful not to overwrite each other's work (a race condition).
• Distributed Memory: Now imagine the same team, but each engineer has their own private notebook. To share an idea, one engineer must write a note and physically pass it to another. The communication is explicit and requires effort from both the sender and receiver.

To build a solid foundation, let's review the formal definitions of these architectures.

The key takeaway is that in a distributed memory system, a process running on one computer (or "node") cannot directly access the memory of a process on another. This lack of a shared address space forces us to rethink how processes can cooperate.

2. The Message-Passing Paradigm

If processes are isolated in their own memory spaces, they must communicate by explicitly sending and receiving data. This is the message-passing paradigm. A process that needs to share data packages it into a message and sends it over a network to a specific destination process. The destination process must then execute a corresponding receive operation to accept the data.

This is fundamentally different from the shared-memory model where communication is implicit—you simply write a value to a shared variable, and other threads can immediately read it (with proper synchronization). In message passing, communication is explicit and cooperative.

To understand this at a conceptual level, let's watch a video that explains inter-process communication (IPC) in general, with a focus on message passing.

This explicit model is the foundation of distributed parallel programming.

The LLNL tutorial you read earlier also has a section that defines this model in the context of high-performance computing. It formally introduces the Message Passing Interface (MPI) as the standard implementation of this model.

3. The SPMD Execution Model

Now that we understand how processes communicate, we need to consider what code they execute. The most common execution model in high-performance computing is SPMD (Single Program, Multiple Data).

As the name suggests, in the SPMD model, every process involved in the computation loads and runs the exact same executable file. This might seem strange at first: if everyone is running the same program, how do they do different work?

The answer lies in conditional logic based on a process's unique identifier, known as its rank. When an MPI program starts, each process is assigned a unique, integer rank, starting from 0. The program can then query its own rank and use it to make decisions.

For example, a common pattern is for process 0 to act as a "manager" or "root" process that distributes work, while all other processes (rank 1, 2, 3, ...) act as "workers".

// Pseudocode for an SPMD program
int my_rank = get_my_rank();
int world_size = get_world_size();

if (my_rank == 0) {
    // I am the manager process.
    // I will prepare the data and send it to the workers.
    for (int i = 1; i < world_size; ++i) {
        send_work_to_process(i);
    }
} else {
    // I am a worker process.
    // I will receive work from the manager process.
    receive_work_from_process(0);
    do_my_computation();
}

All processes execute this same code block, but they take different paths based on their rank.

Let's see the formal definition from the LLNL tutorial.

4. SPMD in Action: A First Look at MPI

Let's tie these concepts together by looking at how a real MPI program is structured. MPI provides a small set of essential functions to manage the SPMD model.

The following video introduces MPI and shows a "Hello, World!" program. This example is the perfect illustration of SPMD: every process runs the same code to print a message, but the message is different for each process because it includes its unique rank.

The video gives you a concrete view of how the abstract concepts of message passing and SPMD are realized in a few lines of code. You don't need to memorize the function syntax just yet; the goal is to understand how they enable the SPMD model.

Conclusion

In this lesson, we transitioned from the shared-memory world of OpenMP to the distributed-memory world of MPI. You learned the foundational concepts that underpin almost all large-scale parallel computing.

Key Takeaways:

• Distributed-Memory Architecture: Composed of independent processes, each with its own private memory. There is no global address space.
• Message-Passing Paradigm: Processes collaborate by explicitly sending and receiving messages to transfer data. This communication is cooperative and managed by the programmer.
• SPMD (Single Program, Multiple Data) Model: The most common execution model for parallel scientific code. All processes run the identical program, but use conditional logic based on their unique rank to perform different tasks.
• MPI (Message Passing Interface): The standard library that provides the functions needed to implement the message-passing/SPMD model, such as getting a process's rank and the total number of processes.

Preview of the Next Lesson

Now that you understand the "what" and "why" of the message-passing model, it's time for the "how". In the next lesson, you will write basic MPI programs using MPI_Init, MPI_Finalize, MPI_Comm_rank, and MPI_Comm_size. You will set up an MPI environment, compile, and run your first parallel application, solidifying the concepts you learned today with hands-on practice.