OpenMP Parallel Loops and Variable Scoping

1. Parallelizing Loops with Work-Sharing

In the previous lesson, every thread in the parallel region executed the entire block of code. To parallelize a loop, we don't want every thread to run every iteration. Instead, we want to split the iterations among the threads. For example, if we have 100 iterations and 4 threads, a simple strategy would be for thread 0 to handle iterations 0-24, thread 1 to handle 25-49, and so on.

OpenMP makes this remarkably simple with a work-sharing construct.

To learn about the syntax, let's start with a reading from a tutorial by Bowdoin College.

As you saw, you can either nest a #pragma omp for inside a #pragma omp parallel block, or you can use the combined #pragma omp parallel for directive, which is more common for simple loop parallelization.

Now, let's see a demonstration. The following video implements a numerical integration example, but the key takeaway is observing how #pragma omp for is applied to a loop to distribute its work.

2. Understanding Variable Scope: shared vs. private

Parallelizing the loop is only half the battle. Now that multiple threads are executing different iterations—potentially at the same time—we must consider how they interact with variables. This is the concept of data scope.

In OpenMP, variables used within a parallel region can be either:

• Shared: A single instance of the variable exists in memory, and all threads can read from and write to it. This is powerful but also dangerous, as it can lead to race conditions.
• Private: Each thread gets its own separate copy of the variable. Operations a thread performs on its private copy are invisible to other threads.

This short video from HPC Education provides a crisp definition of these two categories.

To visualize this, consider the following diagram. Before the parallel region, a single master thread has its own set of variables. When the team of threads is forked, shared variables maintain their single instance, while private variables are duplicated for each thread.

So, how does OpenMP decide which variables are shared and which are private? It follows a set of default rules:

• Variables declared outside the parallel region are shared by default.
• Variables declared inside the parallel region (or a parallel loop) are private by default.
• The loop iteration variable of a parallelized loop is always private.

Relying on these defaults can be risky. To see why, let's watch another segment that demonstrates the chaos that can ensue when a variable that should be private is left as shared.

3. Explicitly Managing Data Scope

To avoid the pitfalls of default scoping, OpenMP provides clauses to let you explicitly define the scope of each variable. Your experience with C++ memory management and variable scope will be very useful here, as the principle is similar: being explicit prevents bugs.

The primary clauses are:

• private(list_of_variables)
• shared(list_of_variables)
• default(none)

The default(none) clause is particularly important. It cancels all default scoping rules and forces you, the programmer, to explicitly declare the scope for every single variable used in the parallel region. While this may seem like extra work, it's a critical safety feature that makes you consciously think about data handling, drastically reducing the chance of race conditions.

This next video provides a comprehensive overview of these clauses and strongly advocates for using default(none).

The syntax for using these clauses is straightforward. The image below shows code examples in both C and Fortran. Notice how private(i2) is added to the pragma to ensure each thread gets its own copy of the i2 variable.

4. Putting It All Together: A Complete Example

Let's solidify these concepts with a complete, correct example: parallel vector addition.
The task is simple: for two vectors a and b, compute a third vector c such that c[i] = a[i] + b[i].

Here is the serial C++ code:

#include <iostream>
#include <vector>

void add_vectors(std::vector<double>& c, const std::vector<double>& a, const std::vector<double>& b) {
    int N = c.size();
    for (int i = 0; i < N; ++i) {
        c[i] = a[i] + b[i];
    }
}

Now, let's parallelize this using our best practices.

1. We'll use the #pragma omp parallel for directive to create threads and distribute the loop iterations in one go.
2. We'll add default(none) to force ourselves to think about every variable.
3. We'll then specify the scope for each variable: a, b, c, N, and i.

Let's analyze the variables:

• a, b, c: These are the vectors. All threads need to access the data within them. Thread 0 needs to write to c[0], thread 1 to c[1], etc. Since each thread operates on a different index, there's no race condition on the elements. The vectors themselves must be shared.
• N: This is the size of the vectors. It's a read-only value. Every thread needs to know the loop bound. It should be shared.
• i: This is the loop counter. Each thread will be working on its own set of indices. Therefore, each thread needs its own private copy of i. The loop iteration variable is automatically private, but with default(none), we must list it. It must be private.

Here is the correct parallel implementation:

#include <iostream>
#include <vector>
#include <omp.h> // Don't forget the OpenMP header

void parallel_add_vectors(std::vector<double>& c, const std::vector<double>& a, const std::vector<double>& b) {
    const int N = c.size();
    
    // N, a, b, and c are captured from the outer scope.



    #pragma omp parallel for default(none) shared(a, b, c, N) private(i)
    for (int i = 0; i < N; ++i) {
        // Each thread executes this line for a different 'i'.
        // Since 'i' is private, and each thread writes to a unique c[i],
        // there is no race condition.
        c[i] = a[i] + b[i];
    }
}

This code correctly and safely parallelizes the vector addition. By explicitly defining the scope of each variable, we have created a program that is both parallel and predictable.

Conclusion

In this lesson, we moved from simple thread creation to practical work-sharing. We learned how to parallelize one of the most fundamental constructs in programming—the for loop—and how to manage the data that flows through it.

Key Takeaways:

• The #pragma omp parallel for directive is a powerful tool for distributing the iterations of a loop across a team of threads.
• Variables in a parallel region are either shared (one copy for all threads) or private (one copy per thread).
• Mismanaging data scope is a primary cause of race conditions. Variables declared outside a parallel region are shared by default.
• Using default(none) is a crucial best practice. It forces you to explicitly define the scope of every variable with the shared() and private() clauses, preventing common parallel programming errors.

Preview of the Next Lesson

Our vector addition example worked smoothly because each thread wrote to a unique memory location (c[i]). But what if all threads need to contribute to a single, final value? For instance, what if we wanted to calculate the sum of all elements in an array? If we use a shared sum variable, we will immediately create a race condition.

In the next lesson, we will tackle this exact problem. You will learn about the reduction clause, an elegant and efficient OpenMP feature for safely performing aggregation operations like sums, products, and finding maximums in parallel.