Distributing and Collecting Data with MPI_Scatter and MPI_Gather

1. Distributing Data Chunks with MPI_Scatter

MPI_Scatter is a one-to-many collective operation. A single root process starts with an array of data, and MPI_Scatter divides this array into contiguous blocks, sending one block to each process (including itself).

The key difference from MPI_Bcast is that each process receives a unique piece of the original data.

The C++ function signature for MPI_Scatter is:

int MPI_Scatter(
    const void* sendbuf,
    int sendcount,
    MPI_Datatype sendtype,
    void* recvbuf,
    int recvcount,
    MPI_Datatype recvtype,
    int root,
    MPI_Comm comm
);

Let's analyze the parameters.

As you've read, here are the crucial points to remember for MPI_Scatter:

• sendbuf: Pointer to the large array on the root process. It's ignored for all other processes.
• sendcount: The number of elements sent to each process. This is not the total number of elements in sendbuf. For an array of size N distributed among P processes, sendcount would be N / P.
• recvbuf: Pointer to a buffer on every process where the received chunk will be stored.
• recvcount: The number of elements each process will receive. This is usually the same as sendcount.
• root: The rank of the process that is doing the scattering.

2. Collecting Data Chunks with MPI_Gather

MPI_Gather is the inverse operation of MPI_Scatter. It performs a many-to-one communication, collecting data from a buffer on each process and concatenating it into a single, larger buffer on the root process. The data is assembled in rank order.

The function signature is identical to MPI_Scatter:

int MPI_Gather(
    const void* sendbuf,
    int sendcount,
    MPI_Datatype sendtype,
    void* recvbuf,
    int recvcount,
    MPI_Datatype recvtype,
    int root,
    MPI_Comm comm
);

Key points for MPI_Gather:

• sendbuf: Pointer to the local data chunk that each process will send.
• sendcount: The number of elements each process is sending.
• recvbuf: Pointer to the large buffer on the root process where all the data will be collected. This buffer must be large enough to hold all the data from all processes. It's ignored for non-root processes.
• recvcount: The number of elements to be received from each individual process. Like sendcount in MPI_Scatter, this is not the total size of recvbuf. It should generally be equal to sendcount.

For a quick video overview of both concepts, the following resource provides a concise explanation.

3. A Complete Scatter/Gather Example

Now let's see how these two functions work together in a complete C++ program. The typical workflow is:

1. Root: Creates a global data array.
2. All: Call MPI_Scatter to distribute the array. The root sends, and all processes (including the root) receive their chunk.
3. All: Each process performs some computation on its local chunk of data.
4. All: Call MPI_Gather to collect the modified chunks. Each process sends its result, and the root receives them all into the original global array.
5. Root: The root process now has the final, fully processed array.

The following resource provides a clean, well-commented C++ code example that follows this exact pattern.

Let's dissect the code from the slides:

Slide 25: Initialization and Allocation

// Master process allocates array and fills it with data.
u = new int[num_proc * num_to_send];
// ... loop to fill u ...

Only the master process (rank 0) allocates and initializes the full source array u. All processes, however, allocate a smaller receive buffer v, which is large enough to hold their individual chunk (num_to_send elements).

Slide 26: The Scatter Operation

// Each process participates in the scatter
MPI_Scatter(u, num_to_send, MPI_INT, 
            v, num_to_send, MPI_INT, 
            MASTER, MPI_COMM_WORLD);

Here, the master process sends num_to_send integers from u to each process. The data is placed into each process's v array.

Slide 26: Local Computation

// Each process, including the master, adds a
// distinguishable value to received data.
for (int i = 0; i < num_to_send; i++) {
    v[i] += 1000 * (my_rank + 1);
}

This is the "work" phase. Each process modifies its local v array. The modification depends on the process's rank, making it easy to verify that the data was correctly distributed and gathered.

Slide 27: The Gather Operation

// Each process participates in the gather.
MPI_Gather(v, num_to_send, MPI_INT,
           u, num_to_send, MPI_INT, 
           MASTER, MPI_COMM_WORLD);

Finally, each process sends its modified v array back to the master. The master collects these chunks and stores them back into the original u array, in rank order. The recvcount is again num_to_send, as it specifies the size of the chunk received from each process.

This example perfectly illustrates the standard "distribute-compute-collect" pattern that you will use frequently in parallel programming.

4. Hands-on Practice: Parallel Array Processing

Now it's your turn to apply these concepts. Your task is to write a C++ program that performs a simple element-wise operation on a large array in parallel.

Your Task:

1. Define a constant for the number of elements per process (e.g., ELEMENTS_PER_PROC = 25).
2. In main, determine the number of processes (world_size) and the current process rank (world_rank).
3. Calculate the total size of the array: total_elements = ELEMENTS_PER_PROC * world_size.
4. On the root process (rank 0) only:
   • Dynamically allocate a float array of size total_elements.
   • Initialize this array with values from 0.0 to total_elements - 1.0.
   • Print the initial array content (or a part of it) for verification.
5. On all processes:
   • Dynamically allocate a local float array of size ELEMENTS_PER_PROC to serve as the receive buffer.
6. Call MPI_Scatter to distribute the data from the root's large array into each process's local buffer.
7. On all processes:
   • Iterate through your local buffer and calculate the square of each element (i.e., local_buffer[i] = local_buffer[i] * local_buffer[i]).
8. Call MPI_Gather to collect the modified local buffers back into the original large array on the root process.
9. On the root process only:
   • Print the final, modified array content (or a part of it) to verify that all chunks were processed and collected correctly.
10. Clean up all dynamically allocated memory.

This exercise will solidify your understanding of the data flow and memory management required for MPI_Scatter and MPI_Gather. Take about 15 minutes to complete it.

Conclusion

Congratulations on completing this lesson! You've added two of the most practical and widely-used MPI collective operations to your toolkit. These functions are the backbone of the "domain decomposition" strategy, where a problem's data is split among processes for parallel computation.

Key Takeaways:

• MPI_Scatter is for one-to-many distribution of distinct data chunks from a root process.
• MPI_Gather is the inverse, a many-to-one collection of distinct chunks back to a root process.
• The sendcount and recvcount parameters specify the number of elements per process, not the total. This is a crucial detail.
• Only the root process needs to allocate the full data buffer for Scatter (as a send buffer) and Gather (as a receive buffer). All processes need to allocate a local buffer for their chunk.

Preview of the Next Lesson

So far, we've learned the building blocks: point-to-point messages, broadcast, reduction, scatter, and gather. In the next lesson, we will use these tools to implement a complete, practical algorithm: parallel matrix-vector multiplication. This will require you to combine data distribution strategies with computation and see how these MPI primitives come together to solve a real-world problem efficiently.