Theorem Deduplication Priority Logic When Merging Multiple Repositories in LeanAgent

LeanAgent resolves duplicate theorems across repositories by keeping the version with the most recent date_processed timestamp, using a canonical key based on file path, theorem name, and source code positions.

When merging theorem data from multiple Lean projects, the lean-dojo/leanagent repository must determine which version of a duplicated theorem to preserve. The theorem deduplication priority logic ensures that the most recently processed data takes precedence while maintaining strict identity criteria for what constitutes a "duplicate."

How Theorem Deduplication Works in LeanAgent

The deduplication process operates in two distinct phases: establishing theorem identity through canonical keys, then applying temporal priority rules to resolve conflicts.

Canonical Key Generation for Theorem Identity

Each theorem receives a unique identifier constructed from its precise location and naming within the source code. In DynamicDatabase.generate_merged_dataset, the system generates a tuple key containing:

key = (
    theorem.file_path,
    theorem.full_name,
    list(theorem.start)[0], list(theorem.start)[1],
    list(theorem.end)[0],   list(theorem.end)[1],
)

This canonical key ensures that two theorems are considered identical only if they share the same file path, fully qualified name, and exact start and end positions (line and column) in the source code.

Timestamp-Based Priority Resolution

Once identity is established, the system applies the most-recent-wins strategy. Each repository carries a metadata["date_processed"] field containing either a datetime object or an ISO-8601 string. The algorithm, implemented in dynamic_database.py (lines 600-614), compares timestamps:

date_processed = repo.metadata["date_processed"]
if isinstance(date_processed, str):
    date_processed = datetime.datetime.fromisoformat(date_processed)

if key not in all_theorems or date_processed > all_theorems[key][1]:
    all_theorems[key] = (theorem, date_processed)

If the current repository's theorem has a newer date_processed value than the existing entry, it replaces the older version in the merged dataset.

Implementation Details in dynamic_database.py

The core deduplication logic resides in the generate_merged_dataset method of the DynamicDatabase class. This method iterates through all repositories, extracts theorems, and builds the all_theorems dictionary using the priority rules described above.

The leanagent.py file invokes this method around line 1103 when constructing unified datasets from multiple Lean project repositories. The repository.py module (embedded within the dynamic database system) defines the metadata structure that stores the critical date_processed timestamps driving the priority decisions.

Summary

• Canonical keys combine file paths, theorem names, and source positions to establish theorem identity.
• Timestamp priority ensures the most recently processed repository version prevails during merges.
• The logic is implemented in DynamicDatabase.generate_merged_dataset within dynamic_database.py (lines 600-614).
• The system handles both datetime objects and ISO-8601 strings for processing dates.

Frequently Asked Questions

What constitutes a duplicate theorem in LeanAgent?

Two theorems are considered duplicates only if they share identical canonical keys, meaning they must have the same file path, fully qualified name, and exact start and end line/column positions in the source code. Theorems with identical names but different locations are treated as distinct entries.

How does LeanAgent handle repositories without date_processed metadata?

The deduplication logic assumes the presence of metadata["date_processed"] for priority comparison. If this field is missing or malformed, the comparison date_processed > all_theorems[key][1] would raise an error or fail to execute properly. Repositories should always include valid ISO-8601 strings or datetime objects in their metadata.

Where is the deduplication logic called in the LeanAgent pipeline?

The generate_merged_dataset method is invoked from leanagent.py around line 1103 when the system needs to consolidate theorem data from multiple repositories into a unified dataset. This typically occurs during the dataset preparation phase before training or analysis begins.

Can the deduplication priority logic be customized?

Currently, the priority logic is hardcoded to use date_processed timestamps with a most-recent-wins strategy. To implement alternative prioritization (such as repository source reliability or theorem proof completeness), you would need to modify the comparison logic in dynamic_database.py within the generate_merged_dataset method.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}