Overlap Communication and Computation with MPI Non-Blocking Calls

1. From Blocking to Non-Blocking Communication

The fundamental difference between blocking and non-blocking communication is when the function call returns.

• Blocking calls (MPI_Send, MPI_Recv) do not return until the communication is complete (or at least, until the application's memory buffer is safe to be reused).
• Non-blocking calls (MPI_Isend, MPI_Irecv) return immediately, initiating the communication and handing it off to the MPI library to manage in the background.

This "fire and forget" approach is the key to unlocking performance, but it requires a new way to track the operation's status.

Let's examine the new functions and the mechanisms for managing them.

As you've just read, every non-blocking operation requires two steps:

1. Initiation: Call MPI_Isend or MPI_Irecv. These functions look very similar to their blocking counterparts, but with one crucial addition: a pointer to an MPI_Request object. MPI populates this object to serve as a handle for the pending operation.
2. Completion Test: At a later point, call MPI_Wait. This function takes the MPI_Request handle and blocks until the corresponding operation has truly finished.

The C++ prototypes are:

// Initiates a non-blocking send
int MPI_Isend(
    void* data,
    int count,
    MPI_Datatype datatype,
    int destination,
    int tag,
    MPI_Comm communicator,
    MPI_Request* request // The new handle
);

// Initiates a non-blocking receive
int MPI_Irecv(
    void* data,
    int count,
    MPI_Datatype datatype,
    int source,
    int tag,
    MPI_Comm communicator,
    MPI_Request* request // The new handle
);

// Waits for a non-blocking operation to complete
int MPI_Wait(
    MPI_Request* request,
    MPI_Status* status
);

The Golden Rule of Non-Blocking Communication

Once you call MPI_Isend, you have promised MPI that you will not modify the send buffer until the operation is complete. Likewise, after calling MPI_Irecv, you must not read from the receive buffer until the operation is complete. The only way to know for sure that an operation is complete is to get a successful return from MPI_Wait.

Violating this rule leads to a race condition and undefined behavior. The data sent could be corrupted, or the data received could be incomplete.

2. Overlapping Communication with Computation

The time gap between initiating the communication (MPI_Isend/MPI_Irecv) and waiting for its completion (MPI_Wait) is where the magic happens. If you can fill this gap with useful computation that doesn't depend on the communication buffers, you have successfully hidden the communication latency.

The following flowchart visualizes this concept perfectly. The ideal case (a) is what we aim for.

Let's look at the code structure that achieves this overlap, adapted from the OxRSE guide.

MPI_Request request;
MPI_Status status;
int important_data[16];
int my_rank;

// Assume my_rank and important_data are set

if (my_rank == 0) {
    // 1. Initiate the send. This returns immediately.
    MPI_Isend(important_data, 16, MPI_INT, 1, 0, MPI_COMM_WORLD, &request);
} else if (my_rank == 1) {
    // 1. Initiate the receive. This also returns immediately.
    MPI_Irecv(important_data, 16, MPI_INT, 0, 0, MPI_COMM_WORLD, &request);
}

// 2. Do other useful work that does NOT touch 'important_data'.
// This computation happens IN PARALLEL with the data transfer.
do_some_independent_calculation(); 

// 3. Wait for the communication to complete.
// If the independent calculation took long enough, this might not block at all!
MPI_Wait(&request, &status);

// 4. Now it's safe to use the data.
// Rank 0 can modify 'important_data' again.
// Rank 1 can read the received values from 'important_data'.
do_something_with_the_data(important_data);

This pattern is the cornerstone of high-performance MPI programming.

3. Non-Blocking Calls and Deadlock Prevention

As a welcome side effect, non-blocking communication can elegantly solve the deadlock problem we discussed in the previous lesson. Recall the deadlock scenario:

• Process 0: MPI_Send(to=1), then MPI_Recv(from=1)
• Process 1: MPI_Send(to=0), then MPI_Recv(from=0)

Both processes get stuck on MPI_Send because neither has posted a matching MPI_Recv.

Now, consider the non-blocking equivalent:

• Process 0: MPI_Isend(to=1), MPI_Irecv(from=1), then MPI_Wait for both.
• Process 1: MPI_Isend(to=0), MPI_Irecv(from=0), then MPI_Wait for both.

Since MPI_Isend and MPI_Irecv return immediately, both processes are able to post both their send and receive requests without blocking. The MPI library can then resolve the pending communications. No deadlock!

4. Alternative Completion: MPI_Test

Sometimes, you might be in a loop where you can do a small piece of work and then check if your communication is done. For this, MPI_Wait is too heavy, as it will block if the communication isn't finished.

The alternative is MPI_Test. It is the non-blocking version of MPI_Wait:

• It checks the status of a request.
• It returns immediately.
• It sets a flag to true if the operation is complete, and false otherwise.

int MPI_Test(
    MPI_Request* request,
    int* flag, // Output: will be true if complete
    MPI_Status* status
);

You can use this in a loop to continue doing work until the communication is finished.

5. Hands-on Practice: A Non-Blocking Ring

It's time to apply what you've learned. In the last lesson, you implemented a ring program using blocking sends and receives. Now, your task is to rewrite it using a more efficient, fully non-blocking pattern.

Your Task:

Modify your ring program to use non-blocking communication. For each process i (including rank 0):

1. Initiate a non-blocking receive (MPI_Irecv) to get a token from process i-1 (with wraparound).
2. Initiate a non-blocking send (MPI_Isend) to send its token to process i+1 (with wraparound).
3. Wait for both the send and receive to complete using MPI_Waitall, which is like MPI_Wait but for an array of requests.
4. After the wait completes, print the message confirming the token was received.

This pattern—posting all receives, then all sends, then waiting for all completions—is a very common and robust way to perform neighbor exchanges in parallel applications like simulations on a grid.

Hint: You will need two MPI_Request objects, one for the send and one for the receive. MPI_Waitall takes an array of these requests.

// Example of using MPI_Waitall
MPI_Request requests[2];
MPI_Status statuses[2];

// ... post Isend and Irecv, storing handles in requests[0] and requests[1]

// Wait for BOTH operations to complete
MPI_Waitall(2, requests, statuses);

Take about 15 minutes to implement this. The following resource contains a complete solution to a very similar problem if you get stuck.

Conclusion

You have now added a critical tool to your parallel programming toolkit. By moving from blocking to non-blocking communication, you can write more sophisticated and performant MPI applications that keep the CPU busy instead of waiting on the network.

Key Takeaways:

• Non-blocking Initiation: MPI_Isend and MPI_Irecv start a communication and return immediately, allowing your program to continue.
• Tracking and Completion: The MPI_Request handle is used to track a pending operation, and MPI_Wait (or MPI_Waitall) is used to ensure it has completed before using the buffers.
• Overlapping for Performance: The primary goal is to perform independent computations between the initiation and completion calls, hiding communication latency.
• Deadlock Avoidance: Non-blocking calls provide a natural way to avoid many common deadlock scenarios.

Preview of the Next Lesson

So far, we have only dealt with point-to-point communication between two processes. However, many parallel algorithms require patterns involving groups of processes, such as broadcasting data from one process to all others, or gathering results from all processes to one. In the next lesson, we will explore collective communication with functions like MPI_Bcast and MPI_Reduce that efficiently implement these group operations.