How LeanAgent Uses LeanDojo's Tracing Functionality to Extract Theorem Data

LeanAgent harnesses LeanDojo's tracer to record tactic executions and transforms the output into structured TracedTheorem objects containing complete proof states.

LeanAgent automatically builds large-scale theorem-proving datasets by leveraging LeanDojo's tracing infrastructure. The repository lean-dojo/leanagent orchestrates the tracing pipeline, captures raw execution data from Lean 4 repositories, and exposes it through Python objects for downstream machine-learning tasks. This article examines the exact mechanism by which LeanAgent extracts theorem data from LeanDojo's tracer output.

The Tracing Pipeline Architecture

LeanAgent does not reimplement tracing logic; instead, it invokes LeanDojo's existing tracer to execute a Lean 4 repository and record every tactic step. The tracer serializes execution details as JSON-Lines files named traced_files.jsonl, which contain file paths and raw dumps of tactic states.

LeanAgent processes this output through three distinct layers:

1. Benchmark generation - Invokes the tracer via generate_benchmark_lean4.main
2. Data persistence - Stores traced_files.jsonl paths in the dynamic database (repo.files_traced)
3. Object materialization - Wraps raw traces in TracedFile and TracedTheorem objects defined in custom_traced_data.py

Step-by-Step Data Extraction Flow

Benchmark Generation and Tracer Invocation

The extraction process begins in leanagent.py (lines 786-795), where the main function calls generate_benchmark_lean4.main(repo.url, sha, dst_dir). This function runs LeanDojo's tracer against the cloned repository and writes one JSON-Line per traced Lean file into dst_dir/traced_files.jsonl.

Each line contains the traced_file_path and the raw XML or JSON representation of every tactic executed in that file. According to the implementation in generate_benchmark_lean4.py (lines 304-314), the tracer captures the full state before and after each tactic application, creating a complete execution trace.

Registering Traced Files in the Database

After the tracer completes, DynamicDatabase.add_repository (located in dynamic_database.py, lines 462-470) persists the path to the JSON-Lines file in the repository metadata under files_traced. This registration allows LeanAgent to reload the trace data without re-running the expensive tracing process.

The database stores these paths as Path objects, enabling direct filesystem access when reconstructing the theorem dataset later.

Building TracedFile and TracedTheorem Objects

The TracedRepo.from_traced_files method in custom_traced_data.py (lines 432-506) reads every line from traced_files.jsonl and constructs TracedFile instances via TracedFile.from_traced_file. This process builds a dependency graph of the files and parses the raw tracer output.

Each TracedFile then generates TracedTheorem objects through get_traced_theorems() (lines 275-329 in custom_traced_data.py). These objects encapsulate:

• Theorem identity: name, full_name, and source location (lean_file, start, end)
• Tactic sequences: A list of TracedTactic objects stored in traced_tactics
• Proof states: state_before and state_after for each tactic step

The TracedTactic objects contain the exact tactic string and the corresponding Lean proof states, providing the raw data required for training theorem-proving models.

Code Example: Extracting Theorem Traces from a Repository

The following Python code demonstrates how to load traced data and iterate over theorem objects:

from leanagent import DynamicDatabase, Repository
from custom_traced_data import TracedRepo

# Load the dynamic database created during benchmark generation

db = DynamicDatabase.from_json("/path/to/dynamic_database.json")

# Select a repository (here the first registered repository)

repo: Repository = db.repositories[0]

# Build a TracedRepo from the recorded traced files

traced_repo = TracedRepo.from_traced_files(
    root_dir=repo.root_dir,                     # Root of the cloned repository

    traced_files_paths=repo.files_traced,       # Paths from traced_files.jsonl

    repo=repo
)

# Iterate over extracted theorems and their tactic traces

for tf in traced_repo.traced_files:            # Each TracedFile

    for thm in tf.get_traced_theorems():       # Each TracedTheorem

        print(f"Theorem: {thm.full_name}")
        print(f"Location: {thm.lean_file}:{thm.start.line_nb}")
        print("Tactic trace:")
        for tac in thm.traced_tactics:
            print(f"  • {tac.tactic}")
            print(f"    Before: {tac.state_before[:60]}...")
            print(f"    After:  {tac.state_after[:60]}...")

This example illustrates how repo.files_traced provides access to the tracer output, while TracedRepo transforms raw JSON-Lines into navigable Python objects. The thm.traced_tactics list contains the precise data captured by LeanDojo, ready for consumption by downstream components like the Retriever or Generator.

Key Source Files and Their Roles

Understanding the codebase structure requires familiarity with these specific modules:

• leanagent.py: Orchestrates the benchmark generation workflow, registers files_traced in the database, and provides high-level entry points for repository processing.
• generate_benchmark_lean4.py: Wraps LeanDojo's tracer execution, handles the export function that writes traced_files.jsonl, and returns the populated TracedRepo.
• custom_traced_data.py: Defines the core data classes TracedFile, TracedTheorem, and TracedTactic; implements the parsing logic that converts tracer JSON into Python objects.
• dynamic_database.py: Persists repository metadata including traced file paths and manages the JSON serialization for the DynamicDatabase.
• retrieval/datamodule.py: Demonstrates downstream consumption patterns, using theorem.traced_tactics to create training datasets for retrieval models.

Summary

• LeanAgent invokes LeanDojo's tracer through generate_benchmark_lean4.main to capture tactic executions as JSON-Lines.
• Trace metadata persists in repo.files_traced within the DynamicDatabase, avoiding redundant computation.
• Raw traces transform into TracedFile objects that parse dependencies and source locations.
• Theorem data materializes as TracedTheorem instances exposing full_name, source spans, and traced_tactics lists.
• Downstream modules consume these objects directly for difficulty estimation, proof search, and model training.

Frequently Asked Questions

How does LeanAgent store the output from LeanDojo's tracer?

LeanAgent stores the path to the traced_files.jsonl file generated by LeanDojo in the repository's files_traced attribute within the DynamicDatabase. The actual trace data remains in the JSON-Lines file on disk, and LeanAgent loads it on demand via TracedRepo.from_traced_files to avoid memory bloat.

What information does a TracedTheorem object contain?

A TracedTheorem object contains the theorem's full_name, source file location (lean_file, start, end), and a complete list of TracedTactic objects in its traced_tactics attribute. Each TracedTactic records the tactic string, the proof state before execution (state_before), and the proof state after execution (state_after).

Can I access theorem traces without regenerating the benchmark?

Yes. Once generate_benchmark_lean4.main completes and DynamicDatabase.add_repository saves the metadata, you can reload the traces by calling DynamicDatabase.from_json followed by TracedRepo.from_traced_files. This uses the stored repo.files_traced paths to reconstruct all TracedTheorem objects without reinvoking the tracer.

Where does the actual parsing of tracer output occur?

The parsing logic resides in custom_traced_data.py, specifically within the TracedFile.from_traced_file class method and the TracedFile.get_traced_theorems method. These functions read the JSON-Lines format, handle XML/JSON deserialization, and instantiate the Python objects that represent theorems and tactics.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}