# Theorem Deduplication Priority Logic When Merging Multiple Repositories in LeanAgent

> Understand LeanAgent's theorem deduplication priority logic when merging repos. Learn how recent date processed and canonical keys ensure the correct theorem version is kept.

- Repository: [LeanDojo/leanagent](https://github.com/lean-dojo/leanagent)
- Tags: deep-dive
- Published: 2026-03-05

---

**LeanAgent resolves duplicate theorems across repositories by keeping the version with the most recent `date_processed` timestamp, using a canonical key based on file path, theorem name, and source code positions.**

When merging theorem data from multiple Lean projects, the lean-dojo/leanagent repository must determine which version of a duplicated theorem to preserve. The theorem deduplication priority logic ensures that the most recently processed data takes precedence while maintaining strict identity criteria for what constitutes a "duplicate."

## How Theorem Deduplication Works in LeanAgent

The deduplication process operates in two distinct phases: establishing theorem identity through canonical keys, then applying temporal priority rules to resolve conflicts.

### Canonical Key Generation for Theorem Identity

Each theorem receives a unique identifier constructed from its precise location and naming within the source code. In `DynamicDatabase.generate_merged_dataset`, the system generates a tuple key containing:

```python
key = (
    theorem.file_path,
    theorem.full_name,
    list(theorem.start)[0], list(theorem.start)[1],
    list(theorem.end)[0],   list(theorem.end)[1],
)

```

This **canonical key** ensures that two theorems are considered identical only if they share the same file path, fully qualified name, and exact start and end positions (line and column) in the source code.

### Timestamp-Based Priority Resolution

Once identity is established, the system applies the **most-recent-wins** strategy. Each repository carries a `metadata["date_processed"]` field containing either a `datetime` object or an ISO-8601 string. The algorithm, implemented in [`dynamic_database.py`](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py) (lines 600-614), compares timestamps:

```python
date_processed = repo.metadata["date_processed"]
if isinstance(date_processed, str):
    date_processed = datetime.datetime.fromisoformat(date_processed)

if key not in all_theorems or date_processed > all_theorems[key][1]:
    all_theorems[key] = (theorem, date_processed)

```

If the current repository's theorem has a newer `date_processed` value than the existing entry, it replaces the older version in the merged dataset.

## Implementation Details in dynamic_database.py

The core deduplication logic resides in the `generate_merged_dataset` method of the `DynamicDatabase` class. This method iterates through all repositories, extracts theorems, and builds the `all_theorems` dictionary using the priority rules described above.

The [`leanagent.py`](https://github.com/lean-dojo/leanagent/blob/main/leanagent.py) file invokes this method around line 1103 when constructing unified datasets from multiple Lean project repositories. The [`repository.py`](https://github.com/lean-dojo/leanagent/blob/main/repository.py) module (embedded within the dynamic database system) defines the metadata structure that stores the critical `date_processed` timestamps driving the priority decisions.

## Summary

- **Canonical keys** combine file paths, theorem names, and source positions to establish theorem identity.
- **Timestamp priority** ensures the most recently processed repository version prevails during merges.
- The logic is implemented in `DynamicDatabase.generate_merged_dataset` within [`dynamic_database.py`](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py) (lines 600-614).
- The system handles both `datetime` objects and ISO-8601 strings for processing dates.

## Frequently Asked Questions

### What constitutes a duplicate theorem in LeanAgent?

Two theorems are considered duplicates only if they share identical canonical keys, meaning they must have the same file path, fully qualified name, and exact start and end line/column positions in the source code. Theorems with identical names but different locations are treated as distinct entries.

### How does LeanAgent handle repositories without date_processed metadata?

The deduplication logic assumes the presence of `metadata["date_processed"]` for priority comparison. If this field is missing or malformed, the comparison `date_processed > all_theorems[key][1]` would raise an error or fail to execute properly. Repositories should always include valid ISO-8601 strings or datetime objects in their metadata.

### Where is the deduplication logic called in the LeanAgent pipeline?

The `generate_merged_dataset` method is invoked from [`leanagent.py`](https://github.com/lean-dojo/leanagent/blob/main/leanagent.py) around line 1103 when the system needs to consolidate theorem data from multiple repositories into a unified dataset. This typically occurs during the dataset preparation phase before training or analysis begins.

### Can the deduplication priority logic be customized?

Currently, the priority logic is hardcoded to use `date_processed` timestamps with a most-recent-wins strategy. To implement alternative prioritization (such as repository source reliability or theorem proof completeness), you would need to modify the comparison logic in [`dynamic_database.py`](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py) within the `generate_merged_dataset` method.