How the Annotation System Links Tactics to Mathematical Premises in Lean Agent

The annotation system in lean-dojo/leanagent records every tactic execution alongside Annotation objects that store fully-qualified names and source locations of mathematical premises, creating a three-layer bridge from low-level tactics to high-level mathematical objects.

The lean-dojo/leanagent repository implements a sophisticated annotation system that links tactics to mathematical premises through structured data objects. This system enables precise tracking of dependencies between proof steps and the mathematical foundations they reference, supporting downstream tasks like premise-aware dataset splitting and model evaluation.

Three-Layer Architecture of the Annotation System

The link between tactics and premises is built through three distinct layers, each defined in dynamic_database.py.

Layer 1: The Annotation Object

Each Annotation stores the fully-qualified name of a premise and its exact source location. According to the source code in dynamic_database.py (lines 54‑60), the class definition captures:

• full_name: The fully-qualified identifier of the lemma, definition, or theorem
• def_path: The file path where the premise is defined
• def_pos and def_end_pos: The start and end positions within the file

The serialization helpers at lines 60‑78 provide from_dict and to_dict methods that convert these objects to and from the JSON schema used in the dataset.

Layer 2: The AnnotatedTactic Class

A tactic is paired with the list of Annotation objects gathered during execution. The AnnotatedTactic class (lines 101‑111 in dynamic_database.py) contains:

• tactic: The raw tactic string
• annotated_tactic: A tuple where the second element (annotated_tactic[1]) holds the list of premise Annotation objects
• state_before and state_after: The proof states surrounding the tactic execution

Conversion helpers at lines 108‑123 manage serialization of the nested annotation data.

Layer 3: Theorem-Level Tracing

Every Theorem stores the complete proof trace as a list of annotated tactics. The Theorem definition (lines 151‑164 in dynamic_database.py) includes the traced_tactics field (lines 159‑160), which contains the list of AnnotatedTactic objects. This structure allows the pipeline to walk from a theorem to its tactics to the specific premises each tactic references.

How the Annotation System Enables Premise-Aware Splitting

The most visible consumer of this linkage is the premise-aware split performed when creating train/validation/test datasets. The _split_by_premise method in dynamic_database.py (lines 78‑96) directly exploits the annotation system:

# In DynamicDatabase._split_by_premise (lines 83-90)

for t in theorems:
    if t.traced_tactics:
        for tactic in t.traced_tactics:
            for annotation in tactic.annotated_tactic[1]:
                theorems_by_premises[annotation.full_name].append(t)

This code walks every traced tactic of every theorem, extracts the annotation.full_name (the premise identifier), and groups theorems by that premise. The surrounding split logic (lines 70‑77) then uses these groups to ensure that validation and test sets contain the fewest premises that also appear in the training set, yielding a realistic "novel-premise" evaluation scenario.

Key Implementation Files and Code Structure

The annotation system is distributed across the following files:

File	Role	Relevant Sections
dynamic_database.py	Core data model (Annotation, AnnotatedTactic, Theorem) and premise-aware split logic	Annotation definition L 54‑L 60, serialization L 60‑L 78; AnnotatedTactic L 101‑L 111, helpers L 108‑L 123; Theorem L 151‑L 164, traced_tactics L 159‑L 160; _split_by_premise L 78‑L 96
unittest_dynamic_database.py	Unit tests constructing annotated tactics and verifying linking	Annotation creation L 215‑L 226; AnnotatedTactic L 228‑L 236
custom_traced_data.py	Helper extracting annotated tactics from traced data	get_annotated_tactic L 174‑L 182
generator/datamodule.py	Serializes annotated tactics for model training	Formatting logic reading annotated_tactic L 70‑L 73
leanagent.py	High-level driver building AnnotatedTactic objects	Instantiation L 619‑L 622

Practical Examples: Working with Annotations

Creating an Annotation and Attaching It to a Tactic

The following example demonstrates creating an Annotation for the Nat.add_comm lemma and attaching it to a rewrite tactic:

from dynamic_database import Annotation, AnnotatedTactic

# Suppose the premise is the lemma `Nat.add_comm`

ann = Annotation(
    full_name="Nat.add_comm",
    def_path="Init/Algebra/Nat.lean",
    def_pos=Pos(123, 4),          # start line/column

    def_end_pos=Pos(124, 1)       # end line/column

)

# The tactic that uses the lemma

tac = AnnotatedTactic(
    tactic="rw [Nat.add_comm]",
    annotated_tactic=("rw [Nat.add_comm]", [ann]),
    state_before="⊢ a + b = b + a",
    state_after="⊢ b + a = a + b"
)

print(tac.annotated_tactic[1][0].full_name)   # → Nat.add_comm

The Annotation structure mirrors the JSON schema used in the dataset (see to_dict at lines 71‑77 in dynamic_database.py).

Embedding the Tactic in a Theorem Trace

To include the annotated tactic in a complete proof trace:

from dynamic_database import Theorem, Path, Pos

thm = Theorem(
    full_name="my_theorem",
    file_path=Path("src/MyFile.lean"),
    start=Pos(10, 1),
    end=Pos(15, 5),
    url="https://github.com/example/repo",
    commit="abc123",
    theorem_statement="∀ a b : ℕ, a + b = b + a",
    traced_tactics=[tac]            # <-- our AnnotatedTactic above

)

When the theorem is serialized (thm.to_dict()), the tactic’s annotated premises become part of the JSON payload, ready for downstream splitters.

Using the Premise-Aware Split

The annotation system enables sophisticated dataset splitting based on premise usage:

db = DynamicDatabase()

# … (populate db with repositories and theorems) …

splits = db._split_by_premise(all_theorems, num_val=100, num_test=100)

# The `val` split will contain theorems that rely on premises

# that appear rarely across the whole corpus.

The core of the split is the loop at lines 84‑90 in dynamic_database.py, which directly accesses annotation.full_name to group theorems by their dependencies.

Summary

• The annotation system creates a three-layer link between tactics and mathematical premises through Annotation, AnnotatedTactic, and Theorem objects.
• Each Annotation stores the fully-qualified name (full_name) and precise source location (def_path, def_pos, def_end_pos) of a premise.
• The AnnotatedTactic class pairs tactic strings with lists of Annotation objects, accessible via annotated_tactic[1].
• The Theorem class aggregates these into traced_tactics, enabling full proof reconstruction with premise dependencies.
• The _split_by_premise method leverages these links to create realistic train/validation/test splits that minimize premise overlap.

Frequently Asked Questions

What information does an Annotation object store?

An Annotation object stores four critical fields defined in dynamic_database.py (lines 54‑60): full_name (the fully-qualified identifier of the premise), def_path (the file path where the premise is defined), and def_pos/def_end_pos (the start and end positions within the file). These fields are serialized via to_dict and deserialized via from_dict (lines 60‑78) to maintain the link between tactics and their mathematical foundations.

How does the annotation system support dataset splitting?

The annotation system enables premise-aware splitting by allowing the _split_by_premise method (lines 78‑96 in dynamic_database.py) to traverse from theorems to their tactics to the specific premises each tactic references. The method extracts annotation.full_name from each AnnotatedTactic and groups theorems by these dependencies, then constructs splits that minimize premise overlap between training and evaluation sets.

Where is the AnnotatedTactic class defined?

The AnnotatedTactic class is defined in dynamic_database.py at lines 101‑111, with serialization helpers at lines 108‑123. The class encapsulates a tactic string, a tuple containing the tactic and its associated Annotation list (annotated_tactic), and the proof states before and after execution. High-level instantiation occurs in leanagent.py (lines 619‑622) when processing Lean proofs.

Can annotations track premises across different files?

Yes, the def_path field in the Annotation class explicitly stores the file path where a premise is defined, enabling cross-file tracking. When the Lean-Dojo extractor parses a Lean file, it records every referenced identifier along with its source location, regardless of whether the premise resides in the current file or an imported module. This allows the annotation system to maintain accurate links to premises defined anywhere in the Lean ecosystem.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}