Crafting Minimal Reproducible Examples for C++ Bugs

The Art of Reduction

A "minimal reproducible example" (MRE) is the gold standard for a bug report. It has several benefits:

• Isolation: It removes all irrelevant code, making it much easier to pinpoint the root cause of the bug.
• Speed: Compiling a tiny file is orders of magnitude faster than building a large project, which dramatically speeds up your debugging cycle.
• Regression Testing: The final MRE becomes the basis for the regression test you will submit with your patch, as you learned to write in the previous lesson.

The overall goal is to transform a situation like "My 10,000-line game engine crashes Clang with -O3" into "This 5-line LLVM IR file crashes opt -passes=gvn." Our focus today will be on compiler crashes, as they are generally more straightforward to reduce than silent miscompilations.

Step 1: From High-Level Code to LLVM IR

Most bugs you'll encounter will not be in the frontend (Clang's C++ parsing) but in the middle-end (optimization) or backend (code generation). Therefore, the first step is almost always to get the LLVM IR representation of the source code and reproduce the crash using LLVM's core tools, opt (the optimizer) and llc (the code generator).

Let's assume a user reports that clang++ -O3 source.cpp -o source crashes.

• If you suspect a backend crash (e.g., the error message mentions a specific machine instruction or register allocation), you want the optimized LLVM IR. You can get this with:

  clang++ -O3 -S -emit-llvm source.cpp -o source.ll
  

  You would then try to reproduce the crash by running llc on the resulting source.ll file.

• More commonly, the crash is in the middle-end during optimization. For this, you need the unoptimized IR that is about to be fed into the optimization pipeline. The correct way to get this is subtle but crucial:

  clang++ -O3 -S -emit-llvm -Xclang -disable-llvm-optzns source.cpp -o source.ll
  

  Here, -O3 tells Clang to set up the full optimization pipeline, but -Xclang -disable-llvm-optzns prevents it from actually running the main LLVM passes. This gives you the exact IR that the opt tool would work on. You would then reproduce the crash by running opt on source.ll.

Step 2: Pinpointing the Crashing Pass

Now you have a (likely very large) .ll file and a command like opt -O3 source.ll -disable-output that crashes. The -O3 pipeline runs hundreds of passes. Which one is the culprit? LLVM provides powerful flags to find out. The goal is to get the IR state just before the single crashing pass.

There are two main techniques to achieve this:

1. -print-on-crash: This is the simplest method. It runs the pipeline and, if a crash occurs, dumps the state of the module or function right before the fatal pass began.

   opt -O3 -print-on-crash -print-module-scope source.ll -disable-output 2> before_crash.ll
   

   The -print-module-scope flag is important; without it, you might only get the single crashing function, which may be useless without its global context. The output will contain the crash backtrace followed by the IR dump. You'll need to manually delete the backtrace from the top of before_crash.ll.

2. -print-pass-numbers and -print-before-pass-number: This two-step process is more efficient for very large files where -print-on-crash might be too slow.

   • First, find the pass number:

     opt -O3 -print-pass-numbers source.ll -disable-output
     

     This will print a log of every pass as it runs, and then crash. The last number printed is the one you want.
   • Then, dump the IR before that pass:

     opt -O3 -print-before-pass-number=N -print-module-scope source.ll -disable-output 2> before_crash.ll
     

     (Where N is the number from the previous step). Again, you'll need to trim the backtrace from the output file.

A newer, cleaner way to get this output is with the -ir-dump-directory flag, which saves the IR to a clean file in a specified directory without the backtrace mixed in.

At the end of this step, you have a .ll file and a single-pass reproducer, for example: opt -passes=gvn -disable-output before_crash.ll.

Step 3: Automatic Minimization with llvm-reduce

You now have a crashing test case, but the before_crash.ll file could still be thousands of lines long. Manually deleting lines of code to find the minimal example is tedious and error-prone. This is where the llvm-reduce tool comes in.

llvm-reduce is a powerful test case reducer. You give it a large input file and a script, called an "interestingness test", that tells it whether a given version of the file is still "interesting" (i.e., still triggers the bug). llvm-reduce then automatically tries thousands of reductions (deleting functions, arguments, instructions, metadata, etc.) and keeps only those that still pass the interestingness test.

For a compiler crash, the interestingness test is a simple shell script. Let's say our single-pass reproducer is opt -passes=gvn -disable-output test.ll. We can create a script named interesting.sh:

#!/bin/bash




# $1 is the temporary file llvm-reduce is testing.
# We want to run our crashing command on it.
# The `!` negates the exit code. A crash gives a non-zero exit code,
# but llvm-reduce considers an exit code of 0 to be "interesting".
! /path/to/your/build/bin/opt -passes=gvn -disable-output "$1" 2>/dev/null

Don't forget to make it executable: chmod +x interesting.sh.

Now, you can run llvm-reduce:

llvm-reduce --test=./interesting.sh before_crash.ll -o reduced.ll

llvm-reduce will now work its magic, potentially for several minutes or hours depending on the complexity, and leave you with a minimal reduced.ll file.

Sometimes, a reduction might uncover a different bug. To be more precise, you can make your interestingness test grep for the specific assertion message from the crash:

#!/bin/bash
/path/to/your/build/bin/opt -passes=gvn -disable-output "$1" 2>&1 | grep -q "Specific Assertion Message"

This script is "interesting" (exits 0) only if the specific error message is found in the output.

A Note on Complex Cases

While this three-step process covers the vast majority of crash reductions, sometimes a bug isn't caused by a single pass but by the interaction between multiple passes (e.g., one pass corrupts some state that a later pass incorrectly uses). A single-pass reproducer won't work here.

In such cases, LLVM provides another tool, reduce_pipeline.py, which works similarly to llvm-reduce but reduces the -passes="..." string itself to find the minimal set of passes that cause the failure. This is an advanced topic, but it's good to know that a tool exists for it.

Conclusion

You have now learned the complete, practical workflow for taking a real-world bug report and boiling it down to its essence. This is a critical skill that separates novice bug-fixers from expert contributors. The process is a systematic funnel: you start wide with a full application and narrow your focus at each stage.

Key Takeaways:

• The goal is a Minimal Reproducible Example (MRE): a small IR file and a single tool invocation.
• Step 1: Use clang with flags like -emit-llvm and -Xclang -disable-llvm-optzns to get the initial IR from a C++ source.
• Step 2: Use opt with -print-on-crash or -print-before-pass-number to isolate the single crashing pass and the IR right before it.
• Step 3: Use llvm-reduce with a shell script interestingness test to automatically minimize the IR file. The script should exit 0 if the bug is present.
• For maximum precision, your interestingness test should grep for the specific crash message.

In this lesson, you saw how llvm-reduce fits into the broader bug triage workflow. In our next lesson, we will take a much deeper look at llvm-reduce itself, exploring its different reduction strategies and more advanced ways to write interestingness tests, solidifying your ability to use this powerful tool.