Profiling and the Observer Effect

Part 1: Setting up Your Benchmark Environment

To run the benchmark we designed, we first need to integrate the Google Benchmark library into our project. The most common and robust way to manage C++ projects and their dependencies is with a build system generator like CMake. While we'll dive deeper into CMake when setting up our GUI in Module 7, for now, we can use a simple configuration to get our benchmark running.

Create a file named CMakeLists.txt in your project's root directory and add the following content:

# Set the minimum required version of CMake and define the project
cmake_minimum_required(VERSION 3.16)
project(TelemetryEngineBench CXX)

# Use the C++17 standard
set(CMAKE_CXX_STANDARD 17)
set(CMAKE_CXX_STANDARD_REQUIRED ON)

# Include the FetchContent module to download external libraries
include(FetchContent)

# Declare the Google Benchmark dependency
FetchContent_Declare(
  GoogleBenchmark
  GIT_REPOSITORY https://github.com/google/benchmark.git
  GIT_TAG        v1.8.3
)

# Make the dependency available to our project
FetchContent_MakeAvailable(GoogleBenchmark)

# Create an executable for our benchmark from a source file
add_executable(OverheadBenchmark benchmark.cpp)

# Link our benchmark executable against Google Benchmark and pthread
target_link_libraries(OverheadBenchmark PRIVATE benchmark pthread)

This file tells CMake to download Google Benchmark and link it against a C++ file we will create called benchmark.cpp. The use of FetchContent makes this process automatic and self-contained.

Part 2: Implementing and Compiling the Benchmark

Now, create the benchmark.cpp file. Copy the full code design from our last lesson into this file. It contains our placeholder macro and the BM_Baseline and BM_Instrumented test cases.

With both CMakeLists.txt and benchmark.cpp in place, open your terminal and follow these steps to compile:

1. Configure CMake (with profiling enabled): This command tells CMake to set up the build files in a build directory. We use the -DPROFILING_ENABLED=1 flag to enable our instrumentation macro.

   cmake -S . -B build -DCMAKE_CXX_FLAGS="-DPROFILING_ENABLED=1"
   

2. Build the executable: This command compiles the code.

   cmake --build build
   

If successful, you will have an executable file named OverheadBenchmark inside the build directory.

Part 3: Running the Benchmark and Quantifying the Observer Effect

It's time for the moment of truth. Run the benchmark from your terminal:

./build/OverheadBenchmark

You will see output similar to this (the exact numbers will vary depending on your computer's hardware):

Run on (12 X 4.50 GHz CPU s)
CPU Caches:
  L1 Data 32 KiB (x6)
  L1 Instruction 32 KiB (x6)
  L2 1024 KiB (x6)
  L3 12288 KiB (x1)
-------------------------------------------------------------------
Benchmark                   Time             CPU         Iterations
-------------------------------------------------------------------
BM_Baseline              151 ns          151 ns            4621981
BM_Instrumented          167 ns          167 ns            4189196

Let's break this down:

• The Time column shows the average wall-clock time it took for one iteration of the for (auto _ : state) loop to complete.
• BM_Baseline took 151 nanoseconds. This is the cost of our DoWork(1000) function call.
• BM_Instrumented took 167 nanoseconds. This is the cost of DoWork(1000) plus the cost of our PROFILE_SCOPE macro.

The overhead—the observer effect—is the difference between these two measurements:

Overhead = Time(BM_Instrumented) - Time(BM_Baseline) = 167 ns - 151 ns = 16 ns

This number is the performance cost of a single instrumentation point when profiling is active. But is 16 ns good? Let's consult a real-world, industry-leading profiler.

Our measurement of 16 ns is remarkably close to Tracy's, confirming that our measurement technique is sound and our initial design is on the right track for high performance.

Part 4: Verifying the "Zero-Overhead" Disabled Case

A key feature of a professional profiler is that it adds zero cost when disabled in a release build. Let's verify our design achieves this.

Re-compile the code, but this time, set the PROFILING_ENABLED flag to 0:

# 1. Re-configure CMake with the flag disabled
cmake -S . -B build -DCMAKE_CXX_FLAGS="-DPROFILING_ENABLED=0"

# 2. Re-build the executable
cmake --build build

Now, run the benchmark executable again:

./build/OverheadBenchmark

The output should now look something like this:

-------------------------------------------------------------------
Benchmark                   Time             CPU         Iterations
-------------------------------------------------------------------
BM_Baseline              151 ns          151 ns            4628205
BM_Instrumented          151 ns          151 ns            4621981

Success! The execution times for BM_Baseline and BM_Instrumented are now identical (or within a negligible margin of system noise). This proves that when our flag is off, the #if PROFILING_ENABLED preprocessor directive works as expected, and the PROFILE_SCOPE macro expands to absolutely nothing, incurring zero performance cost.

This technique is standard practice in high-performance C++.

Conclusion

Congratulations! You have successfully executed a professional-grade microbenchmark and, most importantly, interpreted the results. You've now quantified the observer effect for your placeholder instrumentation and empirically proven that it can be completely removed from a build. This is a crucial milestone for your telemetry engine project.

Key Takeaways:

• You can now use CMake to configure, build, and link a project with an external dependency like Google Benchmark.
• The observer effect is not just a theoretical concept; you measured it to be approximately 16 ns, a figure competitive with industry-standard tools.
• You verified that using preprocessor flags is an effective strategy to achieve true zero-overhead instrumentation when profiling is disabled.
• The process of benchmarking is iterative: compile, run, analyze, and then repeat for a different configuration.

What's Next?

This lesson concludes our deep dive into benchmarking and performance verification. You've built and tested the core data structures and now have a solid methodology for measuring their performance.

We are now ready to begin the final module of the course: Integrating and Polishing the Final Application. In our next lesson, we will begin connecting the pieces. The learning outcome will be: Connect the telemetry capture pipeline's consumer thread to the plotter's data storage. We will finally take the data from our lock-free SPSC queues and feed it into the data structures needed by our ImGui plotter, taking the first step toward visualizing our captured events.