How to Run LLVM Tests: Unit, Regression, and Lit Execution Guide

Run LLVM unit tests with ninja check-llvm-unit, regression tests with ninja check-llvm, and individual tests with llvm-lit <path> after building the project with assertions enabled.

LLVM’s testing infrastructure in the llvm/llvm-project repository is organized into three distinct categories: unit tests, regression tests, and whole-program tests. Knowing how to run LLVM tests efficiently is critical for compiler developers validating optimizations or backend changes. This guide covers the CMake targets, Lit test runner commands, and configuration options required to execute tests at any granularity.

Understanding LLVM's Three Test Categories

LLVM maintains separation between fast unit tests, comprehensive regression tests, and external benchmarking suites.

Unit Tests (Google Test)

Located in llvm/unittests, these are C++ unit tests built on the Google Test framework. They validate individual classes and functions in isolation. According to the source code in llvm/test/Unit/CMakeLists.txt, the check-llvm-unit target is registered via the add_lit_testsuite(check-llvm-unit …) command, which wraps Google Test execution through Lit.

Regression Tests (Lit)

Located in llvm/test, these tests verify end-to-end compiler behavior, code generation, and optimization passes. They are driven by Lit (LLVM’s test runner), implemented in llvm/utils/lit/lit/TestRunner.py. The check-llvm target, defined in llvm/test/CMakeLists.txt using add_lit_testsuite(check-llvm …), executes the full Lit suite.

Whole-Program Test-Suite

The llvm-test-suite repository contains large, whole-program benchmarks and correctness tests that compile and execute external programs. These are not covered here but use similar Lit infrastructure.

Prerequisites: Build LLVM with Assertions

Before running tests, configure a Release build with assertions enabled. This is the recommended configuration for testing, as described in the TestingGuide.md.

Configure the build:

cmake -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=On

Compile the binaries:

ninja -C build

The LLVM_ENABLE_ASSERTIONS=On flag ensures internal consistency checks are active during test execution.

Running LLVM Unit Tests

To execute all unit tests in llvm/unittests, invoke the dedicated CMake target:

ninja -C build check-llvm-unit

If using Makefiles instead of Ninja:

make check-llvm-unit

As implemented in llvm/test/Unit/CMakeLists.txt, this target compiles and runs the Google Test binaries through Lit, reporting granular pass/fail status for individual C++ test cases.

Running LLVM Regression Tests

Regression tests reside in llvm/test and use .ll, .c, .mir, and other file types containing RUN: directives.

Run All Regression Tests

Execute the entire Lit suite using the check-llvm target defined in llvm/test/CMakeLists.txt:

ninja -C build check-llvm

Or with Make:

make check-llvm

Run a Single Test File

For targeted debugging, use the llvm-lit script built alongside the project. The executable resides at build/bin/llvm-lit (or llvm/build/bin/llvm-lit depending on your build directory location):

llvm-lit path/to/llvm-project/llvm/test/Integer/BitPacked.ll

Running a single file is essential when iterating on a specific bug fix or optimization.

Run a Specific Directory of Tests

Point llvm-lit to a directory to execute all tests within it. For example, to run all ARM CodeGen tests:

llvm-lit path/to/llvm-project/llvm/test/CodeGen/ARM

Directories like test/CodeGen/ARM contain a lit.local.cfg file that gates test execution on the availability of the ARM back-end, ensuring tests only run when the corresponding target is compiled into your LLVM build.

Advanced Lit Configuration and Options

Lit provides extensive options for memory checking, filtering, and debugging via the LIT_OPTS environment variable or command-line flags documented in llvm/utils/lit/README.rst.

Valgrind and Memory Checking

Run regression tests under Valgrind for memory error detection:

ninja -C build check LIT_OPTS="-v --vg --vg-leak"

The -v flag enables verbose output, while --vg and --vg-leak invoke Valgrind memory and leak checking.

Single-Threaded Execution

For long-running or flaky tests that require serialized execution to avoid resource conflicts:

ninja -C build check-llvm -j1

The -j1 constraint forces single-threaded execution, often necessary for tests with timing dependencies or hardware simulation requirements.

Key Lit Concepts for Test Authors

Understanding these mechanics helps interpret test failures:

• RUN: lines: Embedded directives in test files that specify how to invoke tools like llvm-as, opt, or FileCheck. Lit parses these lines to construct the test command.
• Substitutions: Placeholders like %s (current file path), %t (temporary file), and %% (escape character) replaced by Lit before execution.
• Constraints: Keywords like REQUIRES, UNSUPPORTED, and XFAIL control whether a test runs based on available features, platforms, or expected failures.
• lit.local.cfg: Per-directory configuration files that define local test configurations, such as required targets or file suffixes.

Summary

• Build configuration: Always use -DLLVM_ENABLE_ASSERTIONS=On when testing.
• Unit tests: Run via ninja check-llvm-unit (defined in llvm/test/Unit/CMakeLists.txt).
• Regression tests: Run via ninja check-llvm (defined in llvm/test/CMakeLists.txt).
• Single tests: Use llvm-lit <file> or llvm-lit <directory> for granular control.
• Advanced options: Use LIT_OPTS for Valgrind and -j1 for single-threaded execution.

Frequently Asked Questions

What is the difference between check-llvm and check-llvm-unit?

check-llvm executes regression tests located in llvm/test using Lit, verifying compiler output and transformation correctness. check-llvm-unit runs C++ unit tests in llvm/unittests using Google Test to validate individual class implementations. Both are implemented as Lit test suites in their respective CMakeLists.txt files.

How do I run a single LLVM test file instead of the entire suite?

Use the llvm-lit executable built in your build directory (e.g., build/bin/llvm-lit) followed by the path to the specific test file in llvm/test. This bypasses the bulk check-llvm target and provides immediate feedback on individual test changes.

What are RUN: lines in LLVM test files?

RUN: lines are directives parsed by the Lit test runner (specifically in llvm/utils/lit/lit/TestRunner.py) that specify the shell commands to execute for a given test. They typically invoke LLVM tools like opt or llc and pipe output to FileCheck for validation.

Why should I enable assertions when building LLVM for testing?

Assertions detect internal compiler inconsistencies and invariant violations during test execution. Without -DLLVM_ENABLE_ASSERTIONS=On, release builds may silently continue past logic errors, masking bugs that the test suite is designed to expose.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}