Skip to main content
Back to path

Welcome to a Grasp lesson

Create a free account to keep studying and create your own study path.

Lesson 15

Running the LLVM Test Suite with llvm-lit

Recommended resources

Welcome to the next step in your journey to becoming an LLVM contributor. In our previous lesson, we demystified the llvm-project directory structure, giving you a map to navigate from a bug report to the relevant source code. You now know that for almost every piece of code in lib/, there's a corresponding test in test/.

Today, we will bring that test/ directory to life. This lesson focuses on the learning outcome: Run the LLVM regression test suite using the llvm-lit testing tool. You'll learn what the LLVM test suite is, how it's organized, and most importantly, how to run it. This is the critical skill that allows you to reproduce a bug, confirm that a bug exists in your local build, and later, verify that your fix has solved the problem without introducing new ones.

The LLVM Testing Ecosystem

Before we run any commands, it's helpful to understand what we're running. The LLVM project employs several types of tests to ensure its quality and correctness.

LLVM Testing Infrastructure Guide — LLVM 23.0.0git documentation

The official LLVM Testing Guide provides an excellent overview of the different categories of tests.

Please read the section that describes the three main types of tests. Focus on understanding the distinction between: Unit tests in llvm/unittests, which use GoogleTest for C++ data structures. Regression tests in llvm/test, which are small code snippets designed to trigger specific behaviors or bugs. Whole-program tests in the separate test-suite repository, used for benchmarking and correctness on larger applications.

For most of your contribution work, especially when starting, you will be interacting almost exclusively with the regression tests in the llvm/test directory. These are the backbone of LLVM's development process.

llvm-lit: The Test Runner

The regression tests are not standalone scripts. They are test files that are discovered and executed by a specialized tool called llvm-lit (LLVM Integrated Tester). lit is a Python-based tool that knows how to find test files, parse special commands within them, execute those commands, and report on the success or failure of the test.

A great way to conceptualize this process is to see it in action. The following video gives a clear, high-level overview of how lit works with a test file.

2019 LLVM Developers’ Meeting: A. Warzynski “Writing an LLVM Pass: 101”

This segment from the "Writing an LLVM Pass: 101" talk provides a concise and practical explanation of lit and its relationship with the test files.

Watch from this section where the speaker introduces lit. Pay close attention to the three core components of a lit test that are described: The IR module (the code being tested). The RUN command, which tells lit how to execute the test. The CHECK directives, which define the expected output.

As you saw, a lit test is essentially a source file (like a .ll file for LLVM IR or a .c file for Clang) containing one or more special comments. The most important of these is the RUN: line.

Anatomy of a Lit Test

The RUN: line is the heart of a lit test. It's a command, or a pipeline of commands, that lit executes. It may look like a shell script, but it is interpreted directly by lit.

Let's look at a typical RUN: line from a test in llvm/test/Transforms/InstCombine/add.ll:

; RUN: opt < %s -passes=instcombine -S | FileCheck %s

Let's break this down:

  • ; RUN:: The comment marker (; for .ll files, // for C++) and the RUN: keyword tell lit that this is a command to execute.
  • opt: This is an LLVM tool. lit automatically substitutes this with the full path to the opt executable you built.
  • < %s: This is input redirection. The %s is a special lit substitution that expands to the full path of the current test file. So, this feeds the test file itself to the opt tool.
  • -passes=instcombine -S: These are command-line arguments for opt, telling it to run the instcombine pass and print the resulting IR in textual form (-S).
  • | FileCheck %s: The | character creates a pipeline. The output of opt is piped as input to the FileCheck tool. FileCheck then verifies this output against CHECK: lines also present in the test file (%s).

lit provides many useful substitutions like %s. Understanding them is key to reading test files.

LLVM Testing Infrastructure Guide — LLVM 23.0.0git documentation

The LLVM Testing Guide documents all the available substitutions. Let's focus on the most common ones you'll encounter.

Please read the Substitutions section of the guide. Focus on understanding these key placeholders: %s: The path to the source file. %t: A path to a temporary file. %S: The directory containing the source file. %clang, %llc, %opt: Substitutions for the paths to the respective tools. %%: How to escape a % character.

How It's All Connected

You might be wondering how lit knows where to find opt, FileCheck, and other tools. This is configured during the build process with CMake. The following diagram illustrates the workflow.

This diagram shows how CMake configures the test environment, which is then used by `ninja check` to run tests via `llvm-lit`.

This diagram shows how CMake configures the test environment, which is then used by ninja check to run tests via llvm-lit.

  1. CMake (Step 1): When you run CMake to configure your build, it takes a template file from the source tree (llvm/test/lit.site.cfg.py.in) and uses it to generate a concrete configuration file (lit.site.cfg.py) in your build folder. This new file contains the full paths to the tools you just built.
  2. ninja check (Step 2): When you run the build target for testing, it invokes llvm-lit.
  3. Configuration Loading (Step 3): lit starts by loading the lit.site.cfg.py file from your build directory. This file, in turn, usually loads the main test configuration (lit.cfg.py) from the llvm/test source directory.
  4. Running Tests (Step 4): Now fully configured, lit begins discovering and running the test cases.

This setup ensures that you are always testing with the exact versions of the tools you just built, avoiding any conflicts with other versions that might be in your system's PATH.

Let's Run Some Tests!

Now for the practical part. Your contribution workflow will rarely involve running all tests at once. Instead, you'll focus on a specific test file or a specific subdirectory related to the bug you're fixing.

All commands should be run from the root of the llvm-project directory.

1. Running All LLVM Tests

This is useful for a final check before submitting a patch, but it's slow. The command is integrated into the build system.

# This command runs all regression tests for core LLVM.
ninja -C build check-llvm

For other sub-projects, you can use targets like check-clang, check-mlir, etc.

2. Running Tests in a Specific Directory

This is far more common. If you are working on a bug in the instruction combination pass, you'll want to run only the tests for that pass. You do this by invoking llvm-lit directly.

# Run all tests in the InstCombine directory
# The -v flag provides verbose output, showing failed tests and their output.
build/bin/llvm-lit -v llvm/test/Transforms/InstCombine/

3. Running a Single Test File

When you're trying to reproduce a specific bug, you'll often narrow it down to a single test file.

# Run a single test file
build/bin/llvm-lit -v llvm/test/Transforms/InstCombine/add.ll

Running this command will execute the RUN: line inside add.ll, pipe the output to FileCheck, and report whether the output matched the CHECK: directives in the file. If it fails, the -v flag will show you the discrepancy, which is exactly what you need to start debugging.

Conclusion

You are now equipped with one of the most fundamental skills for an LLVM contributor: the ability to run the regression test suite. This allows you to verify the existing behavior of the compiler, reproduce reported bugs, and test your own changes.

Key Takeaways:

  • LLVM's regression tests reside in the llvm/test directory and are the primary tool for bug fixing and development.
  • llvm-lit is the test runner that parses test files and executes the commands found in RUN: lines.
  • RUN: lines use special substitutions (like %s for the source file and %opt for the tool) to create portable and self-contained tests.
  • You can run all tests with ninja -C build check-llvm, but it's more common and efficient to run specific subdirectories or files directly with build/bin/llvm-lit <path>.

We've focused on how to run tests that are already written. In our next lesson, we will shift our focus to writing tests. You will learn how to use the FileCheck utility to write pattern-matching assertions, enabling you to create new test cases for the bugs you want to fix.