How LeanAgent Deduplicates Theorems When Merging Repositories in the Dynamic Database

LeanAgent deduplicates theorems during repository merging by generating a unique composite key from file path, fully-qualified name, and exact source location, then retaining only the version with the most recent processing timestamp.

When building training datasets for neural theorem proving, LeanAgent aggregates theorems from multiple Lean repositories and commit histories. To prevent data leakage and redundant training examples, the framework implements a deterministic deduplication strategy within the DynamicDatabase class. This process ensures that each distinct theorem appears exactly once in the final merged output, even when the same theorem exists across different repository versions or forks.

The Deduplication Logic in generate_merged_dataset

The core deduplication mechanism resides in dynamic_database.py within the generate_merged_dataset method. This function orchestrates the merging process by iterating over selected repositories and enforcing uniqueness through a composite key strategy.

Step 1: Collecting Theorems Across Repositories

For each repository selected for merging, the method invokes repo.get_all_theorems, which aggregates proven theorems, sorry-proved theorems, and sorry-unproved theorems into a single collection. This comprehensive enumeration ensures the deduplication logic evaluates every candidate theorem regardless of its proof status.

Step 2: Constructing Unique Identification Keys

LeanAgent creates a deterministic key tuple for each theorem that captures both logical identity and physical location in the source code. As implemented in dynamic_database.py (lines 609-613), the key consists of:

key = (
    theorem.file_path,               # Path of the source file

    theorem.full_name,               # Fully-qualified theorem name

    list(theorem.start)[0],          # Start line

    list(theorem.start)[1],          # Start column

    list(theorem.end)[0],            # End line

    list(theorem.end)[1]             # End column

)

This six-element tuple ensures that two theorems are considered duplicates only if they share the same fully-qualified name and occupy the identical character range within the same source file. The granularity prevents collisions between theorems with similar names in different namespaces or different versions of the same file.

Step 3: Timestamp-Based Conflict Resolution

When duplicate keys occur across repositories or commits, LeanAgent resolves conflicts using repository metadata. Each repository carries a metadata["date_processed"] timestamp indicating when LeanAgent processed that repository version.

The deduplication logic (lines 614-620 in dynamic_database.py) implements a last-write-wins strategy:

if key not in all_theorems or date_processed > all_theorems[key][1]:
    all_theorems[key] = (theorem, date_processed)

This comparison guarantees that when a theorem exists in multiple versions, the iteration from the most recently processed repository overwrites older entries. The timestamp-based selection ensures the dynamic database always contains the freshest theorem definitions and proof states.

Step 4: Finalizing the Deduplicated Dataset

After processing all repositories, the method extracts the deduplicated theorems from the dictionary values:

theorems = [t for t, _ in all_theorems.values()]

The resulting list contains unique theorem objects that subsequently undergo train/validation/test splitting before serialization to disk.

Handling Traced Files and Repository Metadata

Beyond theorem deduplication, generate_merged_dataset manages traced file metadata to prevent redundant file entries. The method aggregates traced files using a Python set:

all_traced_files.update(repo.files_traced)

Because all_traced_files is a set rather than a list, duplicate file paths automatically collapse into single entries. This set-based deduplication complements the theorem-level logic, ensuring the merged dataset maintains clean, non-redundant references to all source files involved in the training data.

Practical Implementation Examples

You can observe or replicate LeanAgent's deduplication behavior using the following patterns.

Manual Deduplication of Theorem Collections

To implement the same deduplication logic outside the standard pipeline:

from datetime import datetime
from typing import List, Dict, Tuple

def deduplicate_theorems(repos: List) -> List:
    """
    Replicate LeanAgent's deduplication strategy.
    repos: List of Repository objects with get_all_theorems and metadata.
    """
    all_theorems: Dict[Tuple, Tuple] = {}
    
    for repo in repos:
        timestamp = repo.metadata["date_processed"]
        if isinstance(timestamp, str):
            timestamp = datetime.fromisoformat(timestamp)
            
        for thm in repo.get_all_theorems:
            # Construct the composite key as in dynamic_database.py

            key = (
                thm.file_path,
                thm.full_name,
                list(thm.start)[0],
                list(thm.start)[1],
                list(thm.end)[0],
                list(thm.end)[1],
            )
            
            # Keep newest version based on processing date

            if key not in all_theorems or timestamp > all_theorems[key][1]:
                all_theorems[key] = (thm, timestamp)
                
    return [t for t, _ in all_theorems.values()]

Using DynamicDatabase for Repository Merging

For standard LeanAgent workflows, instantiate DynamicDatabase and invoke the built-in merging capability:

from pathlib import Path
from dynamic_database import DynamicDatabase

# Initialize database and load repositories

db = DynamicDatabase()
db.add_repository(repo_a)  # Repository object with theorems

db.add_repository(repo_b)  # Another repository or commit

# Execute merge with automatic deduplication

output_dir = Path("/tmp/leanagent_merged")
db.generate_merged_dataset(output_dir)

# Output contains deduplicated theorems and traced files

print(f"Deduplicated dataset written to {output_dir}")

Summary

• Composite Key Strategy: LeanAgent identifies duplicate theorems using a tuple of (file_path, full_name, start_line, start_col, end_line, end_col), ensuring exact source location matching.
• Temporal Resolution: When duplicates exist, the system retains the theorem from the repository with the most recent date_processed timestamp.
• Implementation Location: All deduplication logic resides in dynamic_database.py within the DynamicDatabase.generate_merged_dataset method (lines 609-620).
• File-Level Deduplication: Traced files are deduplicated using Python sets (all_traced_files), preventing redundant file metadata in the output.
• Data Integrity: This approach guarantees that merged datasets contain exactly one instance of each theorem, eliminating training data redundancy while preserving the latest theorem versions.

Frequently Asked Questions

How does LeanAgent determine if two theorems are identical?

LeanAgent considers two theorems identical only if they share the same fully-qualified name (full_name) and occupy the exact same character range in the same source file. The deduplication key explicitly includes file_path, start coordinates (line and column), and end coordinates, ensuring that theorems with identical names but different locations or implementations are treated as distinct entries.

What happens when the same theorem exists in multiple repository commits?

When generate_merged_dataset encounters a theorem key that already exists in the accumulator, it compares the date_processed timestamps from the repository metadata. The theorem from the repository with the more recent processing date overwrites the older entry. This timestamp-based resolution ensures the dynamic database always reflects the most current version of each theorem across all processed commits.

Can I customize the deduplication criteria in LeanAgent?

The deduplication logic is hardcoded in dynamic_database.py within the generate_merged_dataset method. To modify the criteria—such as changing the key components or the conflict resolution strategy—you would need to subclass DynamicDatabase and override this method. The current implementation uses a dictionary-based approach with immutable tuple keys, making it straightforward to extend or modify for custom deduplication rules.

Does LeanAgent deduplicate anything besides theorems?

Yes, the merging process also deduplicates traced files. The method aggregates repo.files_traced from all repositories into a Python set (all_traced_files), which automatically eliminates duplicate file path entries. However, premise data and tactic states are not deduplicated through this specific mechanism; they are managed separately within each repository's internal structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}