Dynamic Database JSON Serialization in LeanAgent: Structure and Non-Serializable Type Handling

The DynamicDatabase class in lean-dojo/leanagent serializes Lean proof repositories to JSON by recursively converting dataclasses to dictionaries via to_dict() methods, transforming non-JSON-native Python objects—including datetime, Path, and Pos—into ISO-8601 strings, plain strings, or coordinate arrays before writing to disk.

The lean-dojo/leanagent repository provides utilities for extracting and managing Lean proof data as structured databases. Understanding the exact JSON schema produced by the DynamicDatabase.to_json method is essential for integrating exported datasets with external verification or machine learning tools. This article details the hierarchical structure of the exported JSON and explains the specific conversion strategies used for Python types that are not natively JSON-serializable.

JSON Structure of the Dynamic Database

The serialization process begins at the DynamicDatabase level and cascades through nested dataclasses via their respective to_dict methods. The resulting JSON follows a strict hierarchy from database to repository to individual theorems and tactics.

Database Root Object

At the top level, the JSON contains a single key mapping to an array of repositories.

{
  "repositories": [ ... ]
}

This structure is generated by DynamicDatabase.to_dict, which iterates over the repositories list and calls Repository.to_dict on each element. See the implementation at lines 71–78 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L71-L78).

Repository Schema

Each repository object contains metadata, statistics, and three categories of theorems. The Repository.to_dict method (lines 74–99 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L74-L99)) produces the following fields:

Field	Type	Description
url	string	Repository remote URL
name	string	Repository name
commit	string	Git commit hash
lean_version	string	Lean toolchain version
lean_dojo_version	string	LeanDojo library version
metadata	object	Contains date_processed (ISO-8601 string)
total_theorems	integer	Count of all theorems
num_proven_theorems	integer	Count of proven theorems
num_sorry_theorems	integer	Count of admitted theorems
proven_theorems	array	List of Theorem objects
sorry_theorems_proved	array	List of proven sorry theorems
sorry_theorems_unproved	array	List of unproved sorry theorems
premise_files	array	List of PremiseFile objects
files_traced	array	List of file path strings (Path objects converted)
pr_url	string or null	Optional pull request URL

Theorem Records

Individual theorems are serialized by Theorem.to_dict (lines 91–101 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L91-L101)). Each theorem object includes positional information and tactic traces.

{
  "full_name": "Theorem.Namespace.name",
  "theorem_statement": "∀ n : ℕ, n + 0 = n",
  "file_path": "src/Mathlib/Data/Nat/Basic.lean",
  "start": "Pos(12, 3)",
  "end": "Pos(15, 1)",
  "url": "https://github.com/...",
  "commit": "abc123",
  "traced_tactics": [ ... ],
  "difficulty_rating": 3.2
}

Note that file_path is stored as a string (converted from Path), while start and end positions use the repr string format "Pos(line, col)".

Tactic Annotations

The traced_tactics array contains AnnotatedTactic objects. Within these objects, positions are handled differently than in the top-level theorem: they are stored as coordinate arrays.

{
  "tactic": "simp",
  "annotated_tactic": [
    "simp",
    [
      {
        "full_name": "Nat.add_comm",
        "def_path": "Nat.lean",
        "def_pos": [10, 5],
        "def_end_pos": [10, 14]
      }
    ]
  ],
  "state_before": "...",
  "state_after": "..."
}

This conversion occurs in _export_proofs (lines 15–27 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L15-L27)), where list(a.def_pos) transforms the Pos object into [line, column]. This works because the external Pos class implements the sequence protocol.

Premise Files and Corpus Export

Premise files follow a similar schema for positions as lists. The PremiseFile.to_dict and Premise.to_dict methods (lines 91–96 and 46–52 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L91-L96)) store premise positions as arrays:

{
  "path": "src/Mathlib/Data/Nat/Basic.lean",
  "imports": ["Init", "Nat"],
  "premises": [
    {
      "full_name": "Nat.add_comm",
      "code": "...",
      "start": [10, 5],
      "end": [10, 14],
      "kind": "theorem"
    }
  ]
}

For large-scale exports, _merge_corpus (lines 36–55 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L36-L55)) writes these objects as a JSONL stream (one JSON object per line) to corpus.jsonl rather than a single monolithic array.

Metadata Export

Repository-level statistics are aggregated separately by _export_metadata (lines 58–74 in [dynamic_database.py](https://github.com/lean-dojo/leanagent/blob/main/dynamic_database.py#L58-L74)). This produces a metadata.json file containing counts, version information, and ISO-8601 formatted timestamps.

Handling Non-Serializable Types

The DynamicDatabase serialization pipeline explicitly converts several Python types that the standard json module cannot encode. These conversions happen inside individual to_dict methods, ensuring that DynamicDatabase.to_json only emits JSON-primitive values.

Python Type	JSON Output	Conversion Method	Location
datetime.datetime	ISO-8601 string	datetime.isoformat()	Repository.to_dict (metadata)
pathlib.Path	String	str(self.file_path)	Theorem.to_dict, Repository.to_dict
Pos (line/column)	String "Pos(x, y)"	repr(self.start)	Repository.to_dict, Theorem.to_dict
Pos (line/column)	Array [x, y]	list(self.pos)	_export_proofs, Premise.to_dict

Datetime Serialization

The date_processed field in repository metadata is a datetime.datetime object. During serialization, it is converted to an ISO-8601 formatted string via the isoformat() method, allowing JSON parsers to reconstruct the timestamp if needed.

Path Objects

All pathlib.Path instances—including theorem file paths and traced file lists—are cast to plain strings using str(path) before being added to the output dictionary. This ensures compatibility with filesystem-agnostic JSON consumers.

Positional Coordinates

The Pos class (defined in the external lean-dojo library) represents line and column numbers. The serialization strategy diverges based on context:

• In Repository and Theorem objects: Positions are serialized using repr(), producing a human-readable string like "Pos(12, 3)". This preserves semantic meaning for debugging.
• In exported proofs and premise files: Positions are converted to lists using list(pos), which yields [12, 3]. This numeric array format is more convenient for machine parsing and coordinate arithmetic.

This dual strategy is intentional: the string format appears in high-level repository summaries, while the array format appears in detailed tactic traces and premise databases where coordinate extraction is common.

Summary

• The top-level JSON structure contains a single repositories array, with each repository holding metadata, statistics, and nested theorem lists.
• Non-serializable types are converted inside to_dict() methods before json.dump() is called, preventing serialization errors.
• datetime objects become ISO-8601 strings, Path objects become strings, and Pos objects become either repr strings ("Pos(l, c)") or coordinate lists ([l, c]) depending on the export context.
• The library produces both monolithic JSON files (for repositories) and JSONL streams (for premise corpora) via to_json and _merge_corpus respectively.

Frequently Asked Questions

What is the root structure of the JSON file produced by DynamicDatabase.to_json?

The root is a JSON object with a single key repositories containing an array of repository objects. Each repository includes metadata, theorem lists, and file traces. This structure is defined in DynamicDatabase.to_dict at lines 71–78 of dynamic_database.py.

Why are position coordinates stored differently in theorems versus tactic annotations?

In Theorem objects, positions use the string representation "Pos(line, col)" for readability. In tactic annotations within traced_tactics, positions are stored as arrays [line, col] to facilitate easier parsing and coordinate extraction by downstream machine learning pipelines. The _export_proofs method explicitly uses list(a.def_pos) to produce the array format.

How does the library handle Python Path objects during export?

All pathlib.Path instances are converted to strings using str(path) inside the respective to_dict methods. This occurs for theorem file paths, repository file traces, and premise file paths, ensuring the output contains only JSON-compatible strings rather than complex path objects.

What format are dates stored in within the metadata?

Dates are stored as ISO-8601 formatted strings (e.g., "2023-10-15T14:30:00"). The conversion happens in Repository.to_dict where metadata["date_processed"] is set to self.metadata.date_processed.isoformat(), allowing standard JSON parsers to interpret the temporal data correctly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}