Profiling Multithreaded Applications with `perf`

The Profiling Mindset: From "Does it work?" to "Does it scale?"

For single-threaded applications, performance analysis is often a straightforward search for the most time-consuming function. In multithreaded applications, the picture is more complex. The primary goal of multithreading is often to achieve scalability—the ability to do more work by adding more threads (and utilizing more CPU cores).

An ideal multithreaded application exhibits linear scaling: if you double the number of threads, you halve the execution time. In reality, this is rarely achieved. Communication and synchronization overhead (like waiting for a mutex) create bottlenecks that prevent perfect scaling. Our job as performance engineers is to find and minimize these bottlenecks.

Introducing perf: The Swiss Army Knife of Linux Profiling

The primary tool for this job on Linux is perf. It's a powerful and versatile command-line utility that interfaces with the kernel's Performance Monitoring Unit (PMU) to collect detailed information about a program's execution.

The perf tool has many commands, but we'll focus on the core workflow for identifying hotspots. This diagram illustrates the general process we'll be following.

Step 1: High-Level Analysis with perf stat

Before diving deep, it's useful to get a high-level summary of an application's performance. The perf stat command runs a program and collects aggregate statistics about various performance events.

Let's look at a practical analysis from an excellent blog post on the topic.

As the article demonstrates, a powerful technique is to run perf stat with varying thread counts and compare the results. Let's break down the key insights from the article's perf stat output:

• Wall Time vs. CPU Time: Notice that for the 8-thread run, the user time (128s) is much larger than the time elapsed (22.7s). This is expected in a multithreaded app; it's the sum of time spent across all CPU cores. (user + sys) / elapsed gives you the average CPU utilization.
• Instruction Overhead: This is a crucial insight. The 8-thread run executed 36% more instructions than the 1-thread run to accomplish the exact same task. Where did these extra instructions come from? They aren't doing useful decoding work. They are a symptom of overhead, most likely threads spinning in a loop waiting for a lock to be released.

perf stat gives us the "what" (there is a scaling problem and overhead) but not the "where". For that, we need to sample the application's execution.

Step 2: Pinpointing Hotspots with perf record and perf report

The core of profiling with perf is a two-step process:

1. perf record: Run the application and collect a profile by taking thousands of "samples". A sample is a snapshot of the program's state (primarily the instruction pointer) taken whenever a specific event occurs, most commonly after a certain number of CPU cycles have passed. This generates a perf.data file.
2. perf report: Analyze the perf.data file and present an interactive, sorted list of functions where the most samples were taken. Functions at the top of this list are our hotspots.

To understand the commands and their output, this reference tutorial is very helpful.

Here's an example of what perf report output looks like.

A Practical Workflow for Multithreaded Profiling

Now, let's combine these tools into the powerful workflow demonstrated in the easyperf.net article to find our multithreaded bottleneck.

The key takeaway from this workflow is the comparative analysis. A function whose relative Overhead grows significantly as you add more threads is almost certainly related to a scalability bottleneck. In the article's example, ed_rec_thread went from a negligible 0.18% with one thread to a massive 17.53% with eight threads. The work being done hasn't changed, but the time spent waiting and fighting with other threads has exploded. The article confirms this by finding a busy-wait loop (a spinlock) inside that function's call tree.

Overhead  Symbol
--------  ----------------
  85.10%  apply_filter
   5.30%  read_image_data
   ...

Overhead  Symbol
--------  ----------------
  45.50%  pthread_mutex_lock
  35.20%  apply_filter
   8.10%  write_tile_to_buffer
   ...

Beyond the Basics: Other perf Techniques

perf is incredibly deep. While the record/report workflow is your primary tool, here are a few other capabilities worth knowing about.

• Per-Thread Analysis: The easyperf.net article briefly mentions this. If you suspect one particular thread is behaving badly, you can use perf report -T to list all threads and then perf report --tid <ID> to analyze only the samples from that specific thread.
• Source Code Annotation: Once you've found a hot function with perf report, you can use perf annotate <function_name> to see a disassembly of the function with the overhead percentage next to each instruction. If your program was compiled with debug symbols (-g), perf can even interleave the original C source code. This lets you go from a hot function to the exact hot loop or line of code.
• Flame Graphs: perf has built-in support for generating Flame Graphs (perf script report flamegraph), a powerful visualization that helps you understand the call stacks of your application at a glance. They provide an intuitive view of where time is being spent.

Conclusion

You now have a foundational method for analyzing the performance of your multithreaded C programs. By moving beyond just measuring total runtime and instead looking at how performance characteristics change with thread count, you can systematically uncover the bottlenecks that prevent your code from scaling.

Key Takeaways:

• Profiling multithreaded applications is about finding scalability bottlenecks.
• The primary perf workflow is to perf record your program's execution and then perf report to analyze the collected samples.
• Functions with high Overhead in the perf report output are your hotspots.
• The most powerful analysis technique is to compare perf reports from runs with different numbers of threads. A function whose overhead percentage explodes as threads are added is a strong indicator of a synchronization bottleneck.

Preview of the Next Lesson:
In this lesson, we saw how a hotspot that gets worse with more threads often points to a synchronization problem like lock contention. Our perf report showed pthread_mutex_lock as a hotspot, but how can we be sure? In the next lesson, we will learn how to measure lock contention to identify synchronization bottlenecks, using advanced perf features to directly prove and quantify the time threads spend waiting for locks.