Profiling C++ Code with gprof and perf

Why Profile? "Don't Guess, Measure."

Before optimizing any code, we must first know where the performance bottlenecks are. It's common for developers to assume they know which part of their code is slow, but reality is often surprising. A famous quote, often attributed to Donald Knuth, states that "premature optimization is the root of all evil." Profiling is the antidote to premature optimization. It is a form of dynamic analysis that measures the behavior of a program as it runs, telling you exactly where it spends its time and resources.

There are two primary approaches to profiling:

1. Instrumentation: The compiler inserts special measurement code directly into your program's executable. When the program runs, this code counts function calls and measures execution time. gprof is a classic example of an instrumentation-based profiler.
2. Sampling: The profiler periodically interrupts the program (e.g., thousands of times per second) and records what the program was doing at that instant (i.e., which function or instruction was executing). By aggregating thousands of these samples, it builds a statistical picture of where the program is spending its time. perf is a modern, powerful sampling profiler.

Let's explore both, starting with the classic gprof.

Tool 1: gprof – The Classic Instrumentation Profiler

gprof (the GNU Profiler) is a simple and widely available tool. Its workflow is straightforward and involves three steps:

1. Compile with profiling flags: You must compile your C++ code using the g++ compiler with the -pg flag. This tells the compiler to inject the necessary instrumentation code.
2. Run your program: Execute the compiled program as you normally would. This run will produce a new file, typically named gmon.out, which contains the raw profiling data.
3. Analyze the data: Run gprof on your executable, and it will automatically find and process gmon.out to generate a human-readable report.

As you saw in the video, the gprof report has two main components:

• The Flat Profile: This is a table of all the functions in your program, sorted by the amount of time spent inside each one. The most important columns are % time (the percentage of total execution time spent in that function) and self seconds (the absolute time spent in the function, excluding time spent in functions it called). This table immediately points you to the "hottest" functions.
• The Call Graph: This section provides a more detailed, function-by-function breakdown, showing which functions called the current function (its parents) and which functions it called (its children). This helps you understand the context of the performance cost.

gprof is a great starting point due to its simplicity, but it has limitations. The instrumentation adds overhead, which can slightly alter the program's behavior, and it only provides function-level granularity.

Tool 2: perf – The Modern Sampling Powerhouse

perf is a much more powerful and versatile tool that is integrated directly into the Linux kernel. As a sampling profiler, it has very low overhead and can provide a system-wide view of performance, even capturing interactions with the kernel itself.

The perf workflow is also simple:

1. Compile with debug symbols: Use the -g flag with g++. This doesn't instrument the code, but it provides perf with the necessary information to map performance events back to your source code functions and line numbers.
2. Record performance data: Run your program under the supervision of perf record. For example: perf record ./my_program. This will execute your program and generate a perf.data file.
3. Analyze the data: Use perf report to open an interactive, text-based user interface where you can explore the collected data.

The output of perf report is a table that looks something like this:

The real power of perf lies in its ability to monitor specific hardware performance counters. These are special registers in the CPU that count events like cache misses, branch mispredictions, and CPU cycles. This allows you to directly measure the impact of the cache-unfriendly patterns we discussed in the last lesson.

Hands-On Practice: Find the Bottleneck

Now it's time to apply what you've learned. Here is a C++ program containing a deliberately inefficient function. Your task is to use both gprof and perf to identify it.

Save the following code as profile_test.cpp:

#include <iostream>
#include <vector>
#include <chrono>

const int SIZE = 512; // Reduced size for quicker testing

// A deliberately slow function that accesses memory inefficiently
void process_data_inefficiently() {
    std::vector<std::vector<int>> matrix(SIZE, std::vector<int>(SIZE, 0));
    
    // Column-major traversal in a row-major language (C++)
    for (int j = 0; j < SIZE; ++j) {
        for (int i = 0; i < SIZE; ++i) {
            matrix[i][j] = i + j;
        }
    }
}

// A function that does very little work
void initialize_system() {
    int x = 0;
    for(int i = 0; i < 1000; ++i) {
        x++;
    }
    std::cout << "System initialized." << std::endl;
}

int main() {
    auto start = std::chrono::high_resolution_clock::now();
    
    initialize_system();
    process_data_inefficiently();
    
    auto end = std::chrono::high_resolution_clock::now();
    std::chrono::duration<double> diff = end - start;
    
    std::cout << "Execution finished in " << diff.count() << " seconds." << std::endl;

    return 0;
}

Activity 1: Profiling with gprof (Estimated time: 10 minutes)

1. Open your Linux terminal.
2. Compile the code with gprof instrumentation:
   g++ -pg -O2 -o profile_test profile_test.cpp
   Note: We use an optimization flag like -O2 because profiling unoptimized code is often not representative of real-world performance.
3. Run the executable:
   ./profile_test
4. Analyze the results with gprof. It's helpful to pipe the output to less to make it scrollable:
   gprof profile_test gmon.out | less
5. Examine the flat profile at the top of the report. Which function consumes nearly 100% of the execution time?

Activity 2: Profiling with perf (Estimated time: 10 minutes)

1. Compile the code with debug symbols:
   g++ -g -O2 -o profile_test profile_test.cpp
2. Record performance data using perf record:
   perf record ./profile_test
   (If you get a permissions error, you may need to run this with sudo or adjust your system's perf_event_paranoid setting.)
3. Analyze the data using perf report:
   perf report
4. You should see an interactive report. The function process_data_inefficiently should be at the top.
5. Bonus: Navigate to the process_data_inefficiently line and press the a key to annotate the source code. perf will show you the source code of the function with the percentage of samples that occurred on each line. You'll see that the vast majority of time is spent on the line matrix[i][j] = i + j; inside the nested loops. This confirms that our strided memory access is the bottleneck, just as we predicted in the last lesson. Press q to exit the annotation view and q again to exit perf report.

Conclusion

In this lesson, you've acquired the fundamental skill of performance profiling. You are no longer "flying blind" but can now scientifically measure your code to find the areas most in need of improvement.

Key Takeaways:

• Profiling is Essential: Always measure before you optimize to focus your efforts effectively.
• gprof is a Simple Start: It uses instrumentation and a 3-step compile-run-analyze workflow to give you a function-level breakdown of where time is spent.
• perf is a Professional's Tool: It uses low-overhead sampling and is capable of monitoring deep hardware events like cache misses, providing line-by-line and even assembly-level insights.
• Workflow is Key: For gprof, compile with -pg. For perf, compile with -g for debug symbols to get source-level annotation.

Preview of the Next Lesson

Now that you can precisely identify performance bottlenecks, the next logical step is to learn how to fix them. In the upcoming lesson, we will begin exploring powerful optimization techniques. Our first topic will be Instruction-Level Parallelism (ILP) and CPU pipelining. You'll learn how modern CPUs execute multiple instructions simultaneously and how you can write code that takes full advantage of this incredible hardware capability.