How LeanAgent Handles Lean Version Compatibility Across Diverse Lean 4 Repositories

LeanAgent ensures Lean version compatibility by reading each repository's lean-toolchain file, validating the version against a supported range (v4.3.0-rc2 to v4.8.0-rc1), and automatically searching commit history to find the most recent compatible commit when necessary.

The lean-dojo/leanagent repository processes hundreds of public Lean 4 projects, each potentially targeting different toolchain versions. To prevent crashes and ensure reproducible builds, the codebase implements a robust three-stage compatibility pipeline that validates versions before any proof search or analysis begins.

Reading the Toolchain Configuration

When LeanAgent encounters a new repository, it first extracts the required Lean version from the lean-toolchain file. The system clones the repository, instantiates a LeanGitRepo object, and retrieves the configuration via repo.get_config("lean-toolchain").

The parsing logic resides in generate_benchmark_lean4.py, specifically within the get_lean4_version_from_config function (lines 27-31). This function applies the regex pattern leanprover/lean4:(?P<version>.+?) to extract the semantic version string from the toolchain declaration.

from generate_benchmark_lean4 import get_lean4_version_from_config

config = repo.get_config("lean-toolchain")
lean_version = get_lean4_version_from_config(config["content"])

Validating Against Supported Version Ranges

After extraction, LeanAgent checks whether the version falls within the acceptable support window. The is_supported_version function in generate_benchmark_lean4.py (lines 33-58) implements this policy with strict boundaries:

• Minimum supported: v4.3.0-rc2
• Maximum supported: v4.8.0-rc1

The function normalizes version strings by stripping the leading "v", splitting into major.minor.patch components, and performing tuple comparison against the defined limits.

from generate_benchmark_lean4 import is_supported_version

if is_supported_version(lean_version):
    print("Version compatible")
else:
    print("Version not supported")

If a repository's current toolchain version exceeds these bounds, LeanAgent marks it as incompatible and triggers the commit search algorithm rather than failing immediately.

Searching for Compatible Commits

When the latest commit uses an unsupported Lean version, leanagent.get_compatible_commit (lines 82-100 in leanagent.py) performs a backward search through the repository history. The function executes a deep git fetch to retrieve full commit history, then iterates from newest to oldest, reconstructing a LeanGitRepo for each candidate and checking its lean-toolchain content against is_supported_version. It returns the first SHA whose toolchain satisfies the constraints.

For frequently processed repositories—including mathlib4, SciLean, and pfr—the system short-circuits this search using hardcoded compatible commits defined in add_repo_to_database and find_and_save_compatible_commits (lines 57-68 in leanagent.py). This optimization avoids redundant network operations for well-known projects.

from leanagent import get_compatible_commit

commit, version = get_compatible_commit("https://github.com/leanprover-community/mathlib4.git")

# Returns the newest compatible SHA and its corresponding Lean version

Integration with the Benchmark Pipeline

Once identified, the compatible (commit, version) pair is stored in the benchmark metadata under the lean_version field. This value propagates through the subsequent retrieval, proving, and curriculum learning stages, ensuring that all downstream operations target a validated Lean toolchain.

# Inside add_repo_to_database()

sha, v = get_compatible_commit(url)
if not sha:
    logger.info(f"Failed to find a compatible commit for {url}")
    return None

# sha and v persisted to DynamicDatabase for pipeline use

Summary

• Toolchain extraction: LeanAgent parses lean-toolchain files using regex matching in generate_benchmark_lean4.py to identify the target Lean version.
• Version validation: Only versions between v4.3.0-rc2 and v4.8.0-rc1 are accepted, enforced by is_supported_version with semantic version parsing.
• Commit resolution: When latest commits are incompatible, the system searches git history backwards via get_compatible_commit to locate the most recent valid commit.
• Optimization: High-traffic repositories like mathlib4 use hardcoded compatible commits to skip repetitive searches.
• Metadata persistence: Validated versions and commit SHAs are stored in benchmark metadata for consistent pipeline execution.

Frequently Asked Questions

What is the supported Lean version range for LeanAgent?

LeanAgent officially supports Lean 4 versions from v4.3.0-rc2 through v4.8.0-rc1, as defined in generate_benchmark_lean4.py. The is_supported_version function explicitly checks that the parsed major, minor, and patch numbers fall within these inclusive bounds.

How does LeanAgent handle repositories like mathlib4 that update frequently?

For high-priority repositories including mathlib4, SciLean, and pfr, LeanAgent maintains hardcoded compatible commits in leanagent.py (lines 57-68). This allows add_repo_to_database to bypass the iterative commit search and immediately use known-good versions.

What happens if no compatible commit exists in a repository's history?

If get_compatible_commit exhausts the repository history without finding a version between v4.3.0-rc2 and v4.8.0-rc1, it returns None. The calling function add_repo_to_database logs the failure and skips the repository, preventing incompatible code from entering the benchmark dataset.

Why does LeanAgent need to parse the lean-toolchain file instead of using lean --version?

Parsing lean-toolchain directly allows LeanAgent to determine version requirements before installing or executing any Lean binaries. This approach prevents crashes from version mismatches and enables the commit-search fallback strategy without needing to provision multiple toolchain environments upfront.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}