Profiling with `perf`

1. Why perf? Beyond Traditional Tools

You're likely familiar with tools like top for monitoring CPU usage. While useful, they have significant limitations. A common problem is identifying the source of CPU load when it's caused by many short-lived processes. Since top samples the process list every few seconds, these fleeting processes can start and finish between updates, leaving you with high CPU usage but no obvious culprit.

This is where perf excels. As a kernel-based sampling profiler, it can be triggered by various events, including a high-frequency timer, to capture a statistical snapshot of what the CPU is executing at any given moment.

To see a demonstration of this exact scenario, please watch the following short video segment.

Core Concepts of perf

perf is not just one tool, but a suite of performance analysis tools built into the Linux kernel source tree. It operates in two primary modes:

1. Counting (perf stat): This mode counts the total occurrences of specific events over the lifetime of a command. It's very low overhead and gives you a high-level summary. For example, you can count the total number of instructions executed or cache misses that occurred.
2. Sampling (perf record): This is the profiling mode. perf is configured to record a "sample" whenever a specific event occurs. A sample typically includes the instruction pointer (what code was running) and the call stack. By aggregating thousands of these samples, perf builds a statistical picture of where the system is spending its time.

Event Sources

The power of perf comes from the wide variety of events it can monitor. These events originate from different levels of the system.

For this lesson, we will focus primarily on two event categories:

• Software Events: Events generated by the kernel, like timer interrupts (cpu-clock), which are used for general CPU usage profiling.
• Hardware Events: Events generated by the CPU's Performance Monitoring Unit (PMU). These are also known as Performance Monitoring Counters (PMCs). They give us direct insight into the microarchitectural behavior of the CPU, such as cache-misses and branch-mispredictions.

2. Getting Started: Prerequisites and perf stat

Before diving in, you'll need to install the perf tools, which are usually packaged separately from the kernel. On Debian/Ubuntu systems, this is typically done with:

sudo apt-get install linux-tools-common linux-tools-$(uname -r)

High-Level Analysis with perf stat

Let's start with perf stat. It's the simplest way to get a performance overview of a command. Let's run it on a command that does a fair amount of work, like find / -name "*.c" > /dev/null.

perf stat find / -name "*.c" > /dev/null

You'll see output similar to this:

 Performance counter stats for 'find / -name "*.c" > /dev/null':

      1,234.56 msec task-clock                #    0.998 CPUs utilized
         5,432      context-switches          #    0.004 M/sec
           123      cpu-migrations            #    0.099 K/sec
        45,678      page-faults               #    0.037 M/sec
   3,987,654,321      cycles                    #    3.230 GHz
   2,876,543,210      instructions              #    0.72  insn per cycle
     543,210,987      branches                  #  440.011 M/sec
      21,098,765      branch-misses             #    3.88% of all branches

       1.236012345 seconds time elapsed

Let's break down the key metrics for our learning outcome:

• cycles & instructions: These are fundamental PMCs. Their ratio gives you Instructions Per Cycle (IPC). Here, it's 0.72. An IPC greater than 1 suggests the CPU is efficiently executing instructions. An IPC less than 1, especially in CPU-bound code, suggests the CPU is often stalled, waiting for data from memory.
• cache-misses: This counter (which you can add with -e cache-misses) tells you how many times the CPU needed data that wasn't in its caches, forcing a slow trip to main memory.
• branch-misses: This shows how often the CPU's branch predictor guessed wrong about the outcome of an if statement or a loop. Each misprediction causes the pipeline to be flushed and refilled, wasting cycles. A high percentage (like the 3.88% here) can be a significant performance drain.

3. Profiling CPU Usage with Timed Sampling

perf stat tells you what happened, but not where in the code it happened. To find the hot spots in your code, we use perf record for sampling. The most common use is timed sampling to profile overall CPU usage.

# Profile the entire system for 10 seconds at 99Hz
sudo perf record -F 99 -ag -- sleep 10

• -F 99: Sample at a frequency of 99 Hertz (samples per second). We use 99 instead of 100 to avoid accidentally synchronizing with periodic system tasks, which could skew the results.
• -a: Sample across all CPUs.
• -g: Capture a call graph (stack trace) with each sample. This is essential for understanding the context of a function call.
• -- sleep 10: The command to profile. Here, we're just sleeping, which means perf will sample whatever else is running on the system.

This command generates a perf.data file. To analyze it, we use perf report.

perf report

This opens a text-based user interface (TUI). You can navigate with the arrow keys. The main view shows a list of functions sorted by "overhead"—the percentage of total samples in which that function appeared.

The Importance of Symbols and Frame Pointers

When you first use perf report, you might see a lot of hexadecimal addresses instead of function names. For perf to translate these addresses into meaningful names, it needs two things:

1. Debug Symbols: Code should be compiled with -g. For kernel symbols, you may need to install a debug package (e.g., linux-image-*-dbgsym).
2. Frame Pointers: Compilers often perform an optimization that omits the frame pointer to free up a register. This breaks the ability of profilers like perf to walk the stack. You should compile your code with -fno-omit-frame-pointer for accurate profiling.

For a comprehensive guide on setting up your environment for proper profiling, please review the following resource.

Visualizing Profiles with Flame Graphs

While perf report is powerful, navigating large text-based call trees can be cumbersome. Flame Graphs, another invention by Brendan Gregg, provide a powerful and intuitive visualization of profiling data.

Generating a flame graph involves piping the output of perf script into a series of scripts from Gregg's FlameGraph repository. This is a key skill for any serious performance engineer.

4. Profiling Hardware Events: A Practical Scenario

Now let's move beyond general CPU usage to microarchitectural issues. When your code has a low IPC, the cause is often poor cache utilization or frequent branch mispredictions. perf lets you profile these events directly.

Instead of sampling on a timer (-F 99), we'll sample on a hardware counter overflow.

• Cache Misses: sudo perf record -e cache-misses -c 10000 -g -- ./my_app
• Branch Mispredictions: sudo perf record -e branch-misses -c 10000 -g -- ./my_app

Here, -e specifies the event to sample, and -c 10000 tells the CPU to generate a sample every 10,000 occurrences of that event. Running perf report on the resulting perf.data file will now show you which functions are responsible for the most cache misses or branch mispredictions.

Let's walk through a concrete scenario to see this in action.

This scenario demonstrates the full power of perf: it doesn't just tell you that your program is slow; it leads you to the exact line of code and the specific microarchitectural reason (cache misses due to poor memory layout access) for the slowness.

5. Interview Scenario

Interviewer: "I have a data processing application that is CPU-bound, but its performance is much lower than expected. Running perf stat on it reveals an Instructions Per Cycle (IPC) of 0.5. How would you use perf to determine the root cause of this poor performance?"

Conclusion

In this lesson, you've been introduced to perf, the Linux kernel's premier performance analysis tool. You've seen that it's far more than just a top replacement; it's a sophisticated tool for deep microarchitectural analysis.

Key Takeaways:

• perf operates in two main modes: perf stat for high-level event counting and perf record for detailed profiling via sampling.
• Profiling CPU usage is typically done with timed sampling (-F), which helps identify which functions are on-CPU most often.
• Profiling cache misses and branch mispredictions is done with event-based sampling (-e), which pinpoints the exact code paths causing microarchitectural stalls.
• Getting accurate and readable results requires compiling with debug symbols (-g) and without omitted frame pointers (-fno-omit-frame-pointer).
• Visualizations like Flame Graphs are invaluable for interpreting complex profiling data.

Preview of the Next Lesson:

While perf is excellent for sampling and finding "hot spots," it's not always the best tool for understanding the exact sequence, timing, and latency of events. In our next lesson, we will explore ftrace, the kernel's built-in tracer. You'll learn how to use ftrace and trace-cmd to trace specific kernel function calls, measure latencies, and understand the intricate interactions between different kernel subsystems.