Synthetic Data Generation Using HTML Templates in OLM OCR: A Complete Technical Guide

OLM OCR generates synthetic training data by extracting random pages from PDFs, rendering them as PNG images, using Claude to generate HTML templates that replicate the original layout, and applying optional augmentations before outputting structured HTML, Markdown, and PDF files.

The allenai/olmocr repository provides an automated pipeline for creating high-fidelity synthetic training corpora without manual annotation. This synthetic data generation process using HTML templates converts existing PDF documents into structured HTML representations that preserve visual layout, typography, and document structure. The pipeline enables the creation of large-scale datasets for training document understanding models by leveraging vision-language models to reverse-engineer PDF layouts into semantic markup.

How the Synthetic Data Pipeline Works

The core pipeline is implemented in olmocr/synth/mine_html_templates.py and executes an eight-stage process to transform PDF documents into synthetic training examples.

PDF Acquisition and Page Selection

The pipeline begins by retrieving source documents from either local storage or S3 buckets. The download_s3_pdf function (lines 53-66 in mine_html_templates.py) handles PDF retrieval, supporting both file system and cloud storage paths.

Page selection uses a deterministic seed derived from the PDF’s SHA-1 hash to guarantee reproducibility. The process_pdf function randomly selects one page from the document using this seeded randomization, ensuring that subsequent runs process identical pages from the same source files.

Image Rendering and HTML Generation

Once a page is selected, the pipeline rasterizes it into a base64-encoded PNG using render_pdf_to_base64png (lines 71-74). This image is then transmitted to the Claude API via generate_html_from_image (lines 75-78), which is defined in olmocr/synth/claude_client.py.

The Claude API analyzes the visual layout and returns semantic HTML markup that replicates the document’s structure, including text blocks, headings, tables, and spatial positioning. This HTML template serves as the ground-truth representation for the synthetic training example.

Optional Data Augmentations

The pipeline supports three optional perturbations defined in olmocr/synth/augmentations.py to increase dataset diversity:

• densify_html: Injects additional markup elements to increase HTML complexity and richness.
• introduce_text_errors: Introduces configurable random typographic errors (spelling mistakes, character substitutions) to simulate OCR noise.
• apply_jpeg_compression: Re-encodes the rendered PDF with random JPEG quality levels to simulate compression artifacts.

These augmentations are applied within the process_pdf function before final output generation.

Output Structure and Metadata Enrichment

The pipeline generates three distinct output formats organized in a specific directory structure under output_dir/:

1. HTML files: Stored under html/<name>/, containing the generated templates.
2. Markdown files: Saved under training/<name>/, combining HTML content with YAML front-matter.
3. PDF renderings: Created under pdfs/<name>/, generated via Playwright to produce single-page PDFs from the HTML templates.

Each output file includes YAML front-matter containing metadata such as the source PDF path, page number, and git commit hash. The pipeline creates soft-links between directories to enable downstream training pipelines to reference source PDFs while consuming Markdown outputs.

Rotation Augmentation for Training Robustness

A secondary script, rotate_html_templates.py, provides geometric augmentation to test model robustness against orientation variations. This script copies an existing synthetic dataset and applies rotations to a configurable percentage of PDFs.

The rotate_pdf function (lines 65-90) performs the actual rotation at 90°, 180°, or 270° angles. Simultaneously, update_frontmatter_rotation (lines 100-130) updates the Markdown front-matter to include is_rotation_valid: false and rotation_correction fields, marking these samples as requiring orientation correction during training.

Running the Synthetic Data Generation Pipeline

Execute the pipeline from the command line using the mine_html_templates module:

python -m olmocr.synth.mine_html_templates \
    --input-dir s3://my-bucket/papers/ \
    --output-dir /tmp/olmocr_synth \
    --name my_synth_set \
    --densify \
    --introduce-text-errors 3 \
    --jpegify

Parameter explanations:

• --input-dir: Source directory containing PDFs (local path or S3 URI).
• --output-dir: Destination for html/, training/, and pdfs/ subdirectories.
• --densify: Enables HTML densification augmentation.
• --introduce-text-errors 3: Injects three random typographic errors per document.
• --jpegify: Applies random JPEG compression to rendered PDFs.

To apply rotation augmentation to an existing dataset:

python -m olmocr.synth.rotate_html_templates \
    --src-dir /tmp/olmocr_synth \
    --dst-dir /tmp/olmocr_synth_rotated \
    --rotation-percent 10 \
    --angle 90

Parameter explanations:

• --src-dir: The dataset directory created by mine_html_templates.py.
• --dst-dir: Output directory for the rotated dataset copy.
• --rotation-percent: Percentage of PDFs to rotate (0-100).
• --angle: Rotation angle (90, 180, or 270 degrees).

Summary

• The synthetic data generation process using HTML templates in OLM OCR automates the conversion of PDF layouts into semantic HTML markup using the Claude API.
• The pipeline in olmocr/synth/mine_html_templates.py handles PDF download, page selection, PNG rendering, HTML generation, and optional augmentations.
• Three augmentation strategies—densification, text error injection, and JPEG compression—increase dataset diversity and robustness.
• Outputs are organized into html/, training/, and pdfs/ directories with YAML front-matter metadata linking the generated content to source documents.
• The rotate_html_templates.py script provides geometric augmentation (90°, 180°, 270° rotations) with automatic front-matter updates for orientation-aware training.

Frequently Asked Questions

What is the purpose of using HTML templates in the synthetic data generation process?

The HTML templates serve as structured ground-truth representations that preserve the visual layout and semantic structure of original PDF documents. By converting rasterized pages into HTML markup via the Claude API, OLM OCR creates training data that captures document topology, typography, and spatial relationships in a machine-readable format suitable for training layout-aware language models.

How does OLM OCR ensure reproducibility when selecting pages from PDFs?

The pipeline derives a deterministic seed from the SHA-1 hash of each PDF file name. This seed initializes the random number generator used for page selection within the process_pdf function, ensuring that the same page is selected from the same PDF across multiple executions, which is critical for debugging and dataset versioning.

What file formats does the synthetic data generation pipeline produce?

The pipeline generates three synchronized output formats: HTML templates containing the generated markup, Markdown files with YAML front-matter metadata, and single-page PDFs rendered via Playwright. These are stored in output_dir/html/<name>/, output_dir/training/<name>/, and output_dir/pdfs/<name>/ respectively, with soft-links maintaining cross-references between formats.

How does the rotation augmentation script handle metadata updates?

When rotate_html_templates.py processes a PDF, it not only rotates the document using rotate_pdf but also invokes update_frontmatter_rotation to modify the YAML front-matter in the corresponding Markdown files. The script adds is_rotation_valid: false and specifies the rotation_correction angle required to restore proper orientation, allowing training pipelines to identify and handle rotated samples appropriately.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}