Point-to-Point Communication with Blocking Send/Recv

1. The Anatomy of MPI_Send and MPI_Recv

Point-to-point communication involves a pair of functions: one to send data and one to receive it. In MPI, these are MPI_Send and MPI_Recv. Their function prototypes have several arguments that give you precise control over the data exchange.

Let's dive into a detailed explanation of these functions and their parameters.

Let's review the C++ function prototypes and their arguments, as you will be using them directly.

MPI_Send

int MPI_Send(
    void* data,
    int count,
    MPI_Datatype datatype,
    int destination,
    int tag,
    MPI_Comm communicator
);

• void* data: A pointer to the memory buffer containing the data you want to send. Since you're experienced with C++, you'll recognize the use of void* to create a generic interface that can handle any data type.
• int count: The number of elements in the buffer, not the number of bytes.
• MPI_Datatype datatype: An MPI constant specifying the type of each element (e.g., MPI_INT, MPI_DOUBLE). MPI uses this to correctly interpret the data on potentially different machine architectures.
• int destination: The rank of the process that will receive the message.
• int tag: An integer you can use to label or differentiate messages. The receiver can request a message with a specific tag.
• MPI_Comm communicator: The group of processes this communication belongs to, which will almost always be MPI_COMM_WORLD for now.

MPI_Recv

int MPI_Recv(
    void* data,
    int count,
    MPI_Datatype datatype,
    int source,
    int tag,
    MPI_Comm communicator,
    MPI_Status* status
);

The first six arguments mirror MPI_Send, but with a receiver's perspective:

• data, count, datatype: Describe the buffer where incoming data will be stored. count here specifies the maximum number of elements the buffer can hold.
• source: The rank of the process you expect the message from. You can use MPI_ANY_SOURCE if you don't know or care.
• tag: The tag of the message you want to receive. MPI_ANY_TAG is also valid here.
• MPI_Status* status: A pointer to a status object. After the receive completes, MPI fills this object with information about the message that was actually received, such as its true source, tag, and size. If you don't need this information, you can simply pass MPI_STATUS_IGNORE.

2. A Master-Worker Example

Now, let's see a practical example. A common pattern in parallel computing is "master-worker," where one process (the master, usually rank 0) distributes tasks to other processes (the workers) and collects their results.

The following video demonstrates exactly this pattern. Rank 0 sends a unique integer to each worker. Each worker squares the integer it receives and sends the result back to rank 0.

This example elegantly shows how MPI_Send and MPI_Recv work in pairs to coordinate a task. The master sends to worker i, and worker i receives from the master. Then, worker i sends back to the master, and the master receives from worker i.

3. Blocking Communication and the Danger of Deadlock

The functions MPI_Send and MPI_Recv are blocking. This is a critical concept with profound implications for how you structure your code.

• A blocking MPI_Recv will "block" (i.e., pause the execution of that process) until the message it is waiting for has fully arrived.
• A blocking MPI_Send will block until the message has been safely copied out of your application's memory buffer, either into a system buffer or across the network. Once MPI_Send returns, you are free to reuse the send buffer.

This blocking behavior ensures that communications are completed correctly, but it also creates a classic trap in parallel programming: deadlock.

Imagine two processes, 0 and 1, that need to exchange data. Consider this flawed logic:

• Process 0: Calls MPI_Send to send data to process 1. Then calls MPI_Recv to get data from process 1.
• Process 1: Calls MPI_Send to send data to process 0. Then calls MPI_Recv to get data from process 0.

Both processes call MPI_Send and block, waiting for the other process to post a matching receive. But neither process can get to its MPI_Recv call because it's stuck in MPI_Send. They will wait forever. This is a deadlock.

The solution is to ensure that for every Send, a matching Recv can be executed. Let's look at a resource that explains this problem and its solution clearly.

As the resource demonstrates, the deadlock is resolved by breaking the symmetry. Instead of all processes doing Send then Recv, you can structure it like this:

• Process 0: MPI_Send(to=1), then MPI_Recv(from=1)
• Process 1: MPI_Recv(from=0), then MPI_Send(to=0)

Now, Process 0's send can complete because Process 1 immediately posts a matching receive. Once that transfer is done, Process 1's send can complete because Process 0 has moved on to its receive call. A common way to implement this for many processes is to have even-ranked processes send then receive, while odd-ranked processes receive then send.

4. Hands-on Practice: The Ring

Now it's time to write some code. A classic MPI exercise is to pass a message around a "ring" of processes. Process 0 sends a value to process 1, process 1 sends it to process 2, and so on, until the last process sends it back to 0.

Your Task:

Write a C++ MPI program that implements this ring communication.

1. Process 0 should initialize an integer token to some value (e.g., 42).
2. Process 0 sends this token to process 1.
3. Every other process i should:
   a. Receive the token from process i-1.
   b. Print a message like: Process <i> received token <value> from process <i-1>.
   c. Send the token to process i+1.
4. The communication should wrap around: the last process (world_size - 1) sends the token back to process 0.
5. Process 0, after its initial send, must wait to receive the token from the last process and print a final confirmation message.

Hint: Be very careful about deadlock! A naive implementation where every process (including 0) tries to receive before sending will deadlock, as process 0 has nothing to receive initially. You need special logic for process 0. Think about the correct sequence of Send and Recv calls for rank 0 versus all other ranks.

Take about 15 minutes to try and implement this. A correct implementation is shown in the MPI Tutorial resource under the "Ring Program" section ([resource ID LINK](https://mpitutorial.com/tutorials/mpi-send-and-receive/)). Refer to it if you get stuck, but try to solve the logic yourself first.

Conclusion

In this lesson, you have mastered the fundamentals of point-to-point communication in MPI. This is a huge leap from our previous lesson and unlocks the ability to write truly collaborative parallel programs.

Key Takeaways:

• Point-to-Point Functions: MPI_Send and MPI_Recv are the core functions for exchanging data between two specific processes.
• Function Parameters: You now understand how to use the arguments data, count, datatype, destination/source, tag, and communicator to control a data transfer.
• Blocking Behavior: Both MPI_Send and MPI_Recv are blocking calls, meaning they pause execution until the communication is safe to complete.
• Deadlock Avoidance: You have learned to identify and prevent deadlock by carefully structuring the order of send and receive operations, ensuring that for every MPI_Send, a corresponding MPI_Recv is posted by the partner process.

Preview of the Next Lesson

While blocking communication is safe and easy to reason about, it can be inefficient. A process might sit idle waiting for a communication to complete when it could be doing other useful work. In the next lesson, we will explore non-blocking communication using MPI_Isend, MPI_Irecv, and MPI_Wait. This advanced technique allows you to overlap communication with computation, a key strategy for maximizing performance in scalable systems.