How LeanAgent Implements Curriculum Learning Strategy with Easy/Medium/Hard Difficulty Categories

LeanAgent implements curriculum learning by calculating theorem difficulty using exponential proof-step counts, categorizing theorems into Easy/Medium/Hard buckets using 33rd and 67th percentile thresholds, and ordering repositories by Easy theorem count to progressively increase training complexity.

The lean-dojo/leanagent repository employs a sophisticated curriculum learning strategy to train theorem-proving agents on formal mathematics. This approach automatically classifies theorems into Easy, Medium, and Hard difficulty categories based on proof complexity, enabling the model to learn from simpler cases before tackling advanced mathematical challenges.

How the Curriculum Learning Strategy Works in LeanAgent

LeanAgent’s curriculum learning pipeline operates in five tightly-coupled stages that transform raw theorem data into a structured training schedule. The system processes each theorem in the database to compute a numerical difficulty score, derives statistical thresholds from the entire corpus, assigns categorical labels, handles edge cases for incomplete proofs, and finally sorts repositories to create a progressive learning path.

This entire workflow is triggered only when the curriculum_learning flag is set to True in the main function, and the resulting sorted repository list is saved to sorted_repos.json for downstream training consumption.

Step-by-Step Implementation of Difficulty Classification

Step 1: Computing Raw Difficulty Scores

The calculate_difficulty function in leanagent.py (lines 842-449) computes a numerical difficulty for each theorem by examining its proof structure. The algorithm applies exponential scaling to the number of proof steps (traced_tactics), calculated as exp(step_count).

Special cases handled during computation:

• Theorems with sorry placeholders: Assigned float('inf') and categorized later as Hard (No proof)
• Theorems with no proof steps: Assigned None for later distribution across all categories

from leanagent import calculate_difficulty

difficulty = calculate_difficulty(theorem)

# → float('inf') if a `sorry` is present,

#   None if no proof steps,

#   exp(number_of_steps) otherwise

Step 2: Deriving Percentile Thresholds

After collecting all finite difficulty values across the entire database, LeanAgent computes statistical cut-offs using NumPy’s percentile function. The system calculates the 33rd and 67th percentiles to create three distinct difficulty bands.

percentiles = np.percentile(all_difficulties, [33, 67])

# percentiles[0] = Easy/Medium boundary

# percentiles[1] = Medium/Hard boundary

This data-driven approach ensures that difficulty categories reflect the actual distribution of proof complexity within the specific mathematical corpus being used.

Step 3: Assigning Difficulty Categories

The categorize_difficulty function (lines 851-562 in leanagent.py) maps numerical difficulty values to categorical labels using the previously computed percentile thresholds:

• Easy: difficulty ≤ 33rd percentile
• Medium: 33rd percentile < difficulty ≤ 67th percentile
• Hard: difficulty > 67th percentile
• Hard (No proof): difficulty = inf (theorems containing sorry)
• To_Distribute: difficulty = None (theorems with no steps)

from leanagent import categorize_difficulty

# Assume we already have percentile cut‑offs from the whole corpus

percentiles = [0.95, 2.30]          # example values

category = categorize_difficulty(difficulty, percentiles)

# Returns "Easy", "Medium", "Hard", "Hard (No proof)" or "To_Distribute"

Step 4: Distributing Unclassified Theorems

Theorems that received None difficulty (indicating no proof steps) cannot be statistically categorized. The system handles these through an even distribution algorithm that splits the "To_Distribute" list across all three concrete difficulty buckets.

The implementation calculates a chunk_size = len(to_distribute)//3 and slices the list to populate Easy, Medium, and Hard categories equally (lines 996-1003 in leanagent.py).

# Distribution logic from leanagent.py

chunk_size = len(to_distribute) // 3
easy_theorems.extend(to_distribute[0:chunk_size])
medium_theorems.extend(to_distribute[chunk_size:2*chunk_size])
hard_theorems.extend(to_distribute[2*chunk_size:])

Step 5: Sorting Repositories for Curriculum Training

The final stage creates the curriculum schedule by ordering repositories based on accessibility. The sort_repositories_by_difficulty function sorts repositories by the count of Easy theorems in descending order (lines 1006-1008 in leanagent.py).

from leanagent import sort_repositories_by_difficulty

sorted_repos, theorem_buckets, perc = sort_repositories_by_difficulty(db)

# `sorted_repos` is a list of Repository objects ordered

# by the number of Easy theorems (most Easy first)

This ensures that training begins with repositories containing the highest density of simple theorems, gradually progressing to more complex material as the model's capability increases.

Key Code Implementation Details

The curriculum learning strategy is implemented primarily in leanagent.py with support from dynamic_database.py. The pipeline activates when curriculum_learning = True in the main function (lines 998-1040).

Critical functions and their locations:

• calculate_difficulty (lines 842-449): Computes exponential difficulty scores and handles sorry placeholders
• categorize_difficulty (lines 851-562): Maps numerical scores to categorical labels using percentile thresholds
• sort_repositories_by_difficulty (lines 864-1008): Orchestrates the full pipeline including distribution of unclassified theorems and final repository sorting

The system persists the curriculum structure to sorted_repos.json, which downstream training loops consume to determine data feeding order.

Summary

• LeanAgent implements curriculum learning by calculating theorem difficulty using exponential scaling of proof step counts in calculate_difficulty.
• Difficulty categories (Easy/Medium/Hard) are determined by 33rd and 67th percentile thresholds computed across the entire theorem corpus.
• Theorems containing sorry placeholders are classified as "Hard (No proof)" with infinite difficulty, while theorems without steps are evenly distributed across all categories.
• Repositories are sorted by descending count of Easy theorems to create a progressive training schedule that starts with simpler material.
• The entire pipeline is controlled by the curriculum_learning flag and persists results to sorted_repos.json for downstream training consumption.

Frequently Asked Questions

How does LeanAgent handle theorems that contain sorry placeholders?

Theorems containing sorry placeholders are assigned a difficulty of float('inf') by the calculate_difficulty function in leanagent.py. During categorization, these theorems are labeled as "Hard (No proof)" and are excluded from the standard Easy/Medium/Hard distribution, effectively treating them as the most challenging edge cases in the curriculum.

What happens to theorems with no proof steps in the curriculum learning pipeline?

Theorems with no proof steps receive a difficulty of None rather than a numerical value. The system collects these into a "To_Distribute" category and then evenly splits them across the Easy, Medium, and Hard buckets by calculating a chunk_size = len(to_distribute)//3 and slicing the list. This ensures that theorems lacking proof data still contribute to all difficulty levels rather than being discarded.

Why does LeanAgent use exponential scaling for proof step counts?

LeanAgent applies exp(step_count) when calculating raw difficulty scores to create a non-linear difficulty curve that better reflects the actual complexity of mathematical proofs. Linear scaling would underestimate the difficulty of long proofs, whereas exponential growth ensures that theorems with substantially more tactics receive proportionally higher difficulty scores, creating clearer separation between Easy, Medium, and Hard categories when percentile thresholds are applied.

How are repositories ordered during curriculum training?

Repositories are sorted by the descending count of Easy theorems using the sort_repositories_by_difficulty function in leanagent.py. The system counts how many theorems in each repository fall into the "Easy" category (≤ 33rd percentile difficulty) and orders repositories so those with the most accessible theorems are processed first. This creates a curriculum where the model encounters simpler mathematical concepts before advancing to repositories with higher densities of Medium and Hard theorems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}