How to Generate Synthetic Training Data with OlmOCR: A Complete Pipeline Guide

OlmOCR generates synthetic OCR training data by converting PDF pages into HTML representations via the Claude API, transforming them into markdown with front-matter metadata, and exporting a JSONL file containing automated verification tests.

The allenai/olmocr repository provides a specialized pipeline for creating high-quality synthetic training datasets for OCR models. By leveraging the Anthropic Claude API to interpret PDF layouts and generate structured HTML representations, the mine_html_templates.py script produces markdown ground truth paired with rigorous test suites. This guide walks through the complete workflow from raw PDFs to training-ready datasets.

Prerequisites and Installation

Before generating synthetic data, install the core OlmOCR dependencies and Playwright browser automation tools.

# Install core dependencies

pip install -r requirements.txt

# Install Chromium for HTML rendering

playwright install chromium

The synthetic data generation requires access to the Anthropic Claude API. Export your API key as an environment variable before running any synthesis scripts:

export ANTHROPIC_API_KEY=sk-your-key-here

Step 1: Prepare Your PDF Input Directory

Create a directory containing the PDFs you wish to convert into synthetic training examples. The pipeline will process every PDF in this location.

mkdir -p data/pdfs

# Copy your source PDFs into this directory

Step 2: Execute the HTML Mining Pipeline

The mine_html_templates.py script in olmocr/synth/mine_html_templates.py drives the entire synthetic data generation process. This script orchestrates PDF rendering, HTML generation via Claude, markdown conversion, and test extraction.

python -m olmocr.synth.mine_html_templates \
    --input_dir data/pdfs \
    --output_dir synthetic_output \
    --name synthetic

PDF to Image Rendering

First, the script converts each PDF page into a base64-encoded PNG using the render_pdf_to_base64png function defined in olmocr/data/renderpdf.py. These images serve as visual input for the Claude API.

HTML Generation and Markdown Conversion

For each rendered page, the generate_html_from_image function sends the image to Claude with a specialized prompt requesting a full HTML representation that mirrors the original layout—including columns, headers, footers, images, mathematical notation, and tables.

The html_to_markdown_with_frontmatter function then processes this HTML using a custom PreserveTablesConverter class that ensures tables remain intact while converting the structure to markdown. This conversion preserves superscripts, subscripts, and tabular data that would otherwise be lost in standard HTML-to-text conversion.

Test Suite Generation

The generate_tests_from_html function creates a comprehensive suite of automated tests including text-presence verification, reading-order validation, table structure checks, and rare-word detection. These tests ensure the synthetic data maintains fidelity to the source PDFs.

Output Directory Structure

After execution, the synthetic_output directory contains:

• bench_data/pdfs/synthetic/ – Symlinks to the original PDF files
• bench_data/claude_original/synthetic/ – Generated markdown files with YAML front-matter
• synthetic.jsonl – JSONL file containing all test cases and metadata

Step 3: Augment with Rotation (Optional)

To increase dataset diversity and improve model robustness, use rotate_html_templates.py to apply rotation augmentation to a percentage of your synthetic data.

python -m olmocr.synth.rotate_html_templates \
    --input_dir synthetic_output \
    --output_dir synthetic_augmented \
    --rotation_percentage 5.0

This script copies the entire directory tree, rotates the selected PDFs using pypdf, and updates the corresponding front-matter fields to is_rotation_valid: false and rotation_correction with the applied rotation angle.

Step 4: Load Data for Training

Feed the generated synthetic.jsonl directly into the OlmOCR training pipeline using the BaseMarkdownPDFDataset class located in olmocr/train/dataloader.py.

from olmocr.train.dataloader import BaseMarkdownPDFDataset
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained("Qwen2-VL-7B")

dataset = BaseMarkdownPDFDataset(
    root_dir="synthetic_output",
    pipeline_steps=processor,
)

print(f"Loaded {len(dataset)} synthetic training samples")

The dataset class reads the markdown files and associated test JSONL, tokenizing the content for model consumption. The train.py entry point in olmocr/train/train.py orchestrates the full training loop and accepts any BaseMarkdownPDFDataset instance as input.

Complete End-to-End Workflow

Combine all steps into a single automation pipeline:

# 1. Install dependencies

pip install -r requirements.txt
playwright install chromium

# 2. Configure API access

export ANTHROPIC_API_KEY=sk-...

# 3. Generate synthetic data

python -m olmocr.synth.mine_html_templates \
    --input_dir data/pdfs \
    --output_dir synthetic_output \
    --name synthetic

# 4. Apply rotation augmentation (optional)

python -m olmocr.synth.rotate_html_templates \
    --input_dir synthetic_output \
    --output_dir synthetic_augmented \
    --rotation_percentage 5

# 5. Begin training

python -m olmocr.train.train --config olmocr/train/configs/example_config.yaml

Summary

• Synthetic data generation in OlmOCR relies on mine_html_templates.py to convert PDFs into HTML via the Claude API, then into markdown with front-matter.
• Core utilities include render_pdf_to_base64png in olmocr/data/renderpdf.py for image generation and PreserveTablesConverter for maintaining table structure.
• Automated testing is handled by generate_tests_from_html, creating JSONL entries that validate text presence, ordering, and tabular integrity.
• Data augmentation via rotate_html_templates.py adds rotational diversity by modifying a configurable percentage of PDFs.
• Training integration uses BaseMarkdownPDFDataset from olmocr/train/dataloader.py to consume the synthetic JSONL and markdown files.

Frequently Asked Questions

What file formats does OlmOCR require for synthetic data generation?

OlmOCR accepts standard PDF files as input for the synthetic generation pipeline. The mine_html_templates.py script processes these through Playwright's Chromium browser to generate PNG representations, then outputs markdown files with YAML front-matter and a JSONL file containing the test suite metadata.

How does the pipeline handle complex document layouts like tables and mathematical notation?

The PreserveTablesConverter class in mine_html_templates.py specifically preserves table structures during HTML-to-markdown conversion, ensuring that rows, columns, and cell alignments remain intact. For mathematical notation, the Claude API generates HTML that includes superscripts and subscripts, which the converter maintains in the final markdown output alongside the visual layout information.

Can I customize the percentage of documents receiving rotation augmentation?

Yes, the rotate_html_templates.py script accepts a --rotation_percentage parameter that accepts float values between 0 and 100. The default is 5.0 percent, but you can adjust this based on your robustness requirements. This percentage determines what fraction of the PDFs are randomly selected for rotation using pypdf's rotation capabilities.

What specific tests are included in the generated synthetic.jsonl file?

According to the implementation in mine_html_templates.py, the generate_tests_from_html function creates several verification tests: text-absence checks to ensure no content was lost during conversion, ordering tests to verify reading sequence matches the original layout, table tests to validate row/column integrity, and rare-word tests to confirm unique vocabulary appears correctly in the output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}