Structure and Categories of the olmOCR-Bench Benchmark Suite: A Complete Guide

The olmOCR-Bench benchmark suite evaluates document-level OCR systems through seven distinct document types and five rigorous test classes, using JSON annotations stored in olmocr/bench/sample_data/ and executed via benchmark.py.

The olmOCR-Bench benchmark suite is the standardized evaluation framework for the allenai/olmocr repository, designed to rigorously test how well OCR pipelines handle complex single-page PDF documents. It combines curated document corpora with automated test generation to measure accuracy across math formulas, tables, reading order preservation, and text extraction fidelity.

Document Categories: The Seven Source Types

The olmOCR-Bench suite organizes its corpus into seven distinct document types, each targeting known failure modes of OCR systems. According to the repository README (lines 220-236), these categories are:

• ArXiv Math (AR): Recent mathematics papers sourced from arXiv. These single-TeX source PDFs challenge OCR systems with complex LaTeX formulas. Candidate pages are identified via OCR, matched back to source, validated with KaTeX, and manually verified.

• Old Scans Math (OSM): Public-domain scanned mathematics textbooks from the Internet Archive. Formula-containing pages are identified by OCR, with each formula manually annotated to create ground truth.

• Tables (TA): PDFs containing tabular data drawn from the same internal crawl used for olmOCR-Mix. Gemini-Flash-2.0 selects random cells, generates relational queries, and manual review ensures correctness.

• Old Scans (OS): Historical letters and typewritten documents with existing human transcriptions from the Library of Congress. Scripts produce natural-reading-order text pairs, with header and footer cases added and human-reviewed.

• Headers Footers (HF): Regions identified by DocLayout-YOLO on the internal crawl. Visual masking combined with Gemini-Flash-2.0 extracts text; humans prune false positives and set search-area limits.

• Multi Column (MC): Multi-column page layouts sampled visually from the internal PDF pool. Claude-Sonnet renders pages to HTML, extracts adjacent text segments, and hand-checking validates the reading order.

• Long Tiny Text (LTT): Dense, small-print pages such as dictionary entries and reference lists. Gemini-Flash-2.0 creates test cases followed by manual verification.

These document types are stored in olmocr/bench/sample_data/pdfs/, with corresponding test specifications in olmocr/bench/sample_data/olmocr_pipeline/ and a master index in olmocr/bench/sample_data/dataset.jsonl.

Test Classes: Evaluation Methodology

Each benchmark item falls into one of five test classes that dictate how OCR output is judged, as defined in the README starting at line 53:

• Text Presence: Verifies that a short snippet (1-3 sentences) appears somewhere in the output. Uses soft/fuzzy matching and supports optional "first N/last N characters" constraints.

• Text Absence: Ensures that a given string does not appear in the output, critical for detecting header/footer noise injection. Applies the same fuzzy matching rules as Text Presence.

• Natural Reading Order: Confirms that two blocks of text appear in the correct relative sequence. Allows swapping of whole sections when order is non-critical.

• Table Accuracy: Checks that a specific cell exists and that its neighbors satisfy specified constraints. Supports Markdown tables, though full scoring requires HTML tables to properly validate rowspan and colspan attributes.

• Math Formula Accuracy: Renders a LaTeX expression with KaTeX and verifies that the OCR output contains the same symbolic sequence (order-insensitive). Equations must be wrapped in $...$, $$...$$, \( ... \), or \[ ... \] delimiters.

The evaluation logic for these classes is implemented in olmocr/bench/benchmark.py, which normalizes Unicode and performs the specific checks for each test type.

olmOCR-Bench Architecture and Key Components

The structure of the benchmark suite comprises several interconnected components:

Data Storage and Layout

The dataset resides in olmocr/bench/sample_data/. PDF pages live in sample_data/pdfs/, while JSON annotation files containing the test specifications are located in sample_data/olmocr_pipeline/. The master file sample_data/dataset.jsonl indexes all test cases across the seven document categories.

Conversion Pipeline

olmocr/bench/convert.py transforms raw PDFs into the markdown format required by the benchmark. For users running the full olmOCR pipeline, olmocr/bench/scripts/workspace_to_bench.py adapts the pipeline workspace output to the benchmark layout.

Execution Driver

olmocr/bench/benchmark.py serves as the core evaluation engine. It loads JSON annotations, runs the selected OCR tool on each PDF, normalizes the output, and evaluates every test case according to its designated class.

Reporting and Analysis

olmocr/bench/report.py aggregates per-category scores and produces consumable HTML and LaTeX summaries. For interactive debugging, olmocr/bench/review_app.py launches a Flask-based web UI on a specified port to inspect individual questions and expected answers.

OCR Runner Scripts

The olmocr/bench/runners/ directory contains ready-to-use scripts (e.g., run_mistral.py, run_marker.py, run_paddleocr.py) demonstrating how to benchmark specific OCR backends against the suite.

Running the olmOCR-Bench Benchmark

Execute the following workflow to evaluate an OCR system against the benchmark suite.

Installation and Setup

Install the benchmark dependencies and Playwright for KaTeX rendering:

conda create -n olmocr python=3.11
conda activate olmocr
git clone https://github.com/allenai/olmocr.git
cd olmocr
pip install -e .[bench]
playwright install chromium

Downloading the Dataset

Retrieve the benchmark corpus from Hugging Face:

huggingface-cli download \
    --repo-type dataset \
    --resume-download allenai/olmOCR-bench \
    --local-dir ./olmOCR-bench

Converting Documents

Convert existing PDFs to the required markdown format using the generic converter:

python -m olmocr.bench.convert olmocr_pipeline \
    --dir ./olmOCR-bench/bench_data

Alternatively, process raw PDFs through the main pipeline and adapt the workspace:

python -m olmocr.pipeline ./localworkspace \
    --markdown \
    --pdfs ./olmOCR-bench/bench_data/pdfs/**/*.pdf

python -m olmocr.bench.scripts.workspace_to_bench \
    localworkspace/ \
    olmOCR-bench/bench_data/olmocr \
    --bench-path ./olmOCR-bench/

Executing the Benchmark

Run the evaluation driver to compute scores across all categories:

python -m olmocr.bench.benchmark \
    --dir ./olmOCR-bench/bench_data

The script prints per-category pass percentages and an overall aggregate score.

Interactive Review

Launch the review application to inspect specific test cases:

python -m olmocr.bench.review_app \
    --port 5000 \
    --debug ./olmOCR-bench/bench_data/multi_column.jsonl \
    --force

Access the interface at http://localhost:5000 to explore questions and ground truth annotations.

Summary

The olmOCR-Bench benchmark suite provides a comprehensive framework for evaluating document OCR systems through its dual-categorization approach:

• Seven document types (AR, OSM, TA, OS, HF, MC, LTT) targeting specific PDF complexity domains
• Five test classes (Text Presence, Text Absence, Natural Reading Order, Table Accuracy, Math Formula Accuracy) providing rigorous evaluation criteria
• Modular architecture with clear separation between data storage (sample_data/), conversion tools (convert.py), execution (benchmark.py), and reporting (report.py)
• Practical tooling including runner scripts for major OCR engines and a web-based review application

Frequently Asked Questions

What file formats does olmOCR-Bench require?

The benchmark requires PDF documents for input and markdown files for OCR output evaluation. Annotations are stored as JSON Lines (.jsonl) files in olmocr/bench/sample_data/olmocr_pipeline/, with a master index at sample_data/dataset.jsonl. The conversion pipeline in olmocr/bench/convert.py handles the transformation from raw PDF to the required markdown format.

How does the benchmark handle math formula verification?

The Math Formula Accuracy test class uses KaTeX to render LaTeX expressions from the ground truth and compares them symbolically against the OCR output. According to the source code, equations must be delimited by $...$, $$...$$, \( ... \), or \[ ... \]. The comparison is order-insensitive for symbols but requires complete symbolic sequence matching, implemented in the evaluation logic of olmocr/bench/benchmark.py.

Can I benchmark custom OCR pipelines not included in the runners?

Yes. Use olmocr/bench/convert.py to transform your OCR output into the benchmark's expected markdown format, or manually place markdown files (.md) alongside the JSON annotation files in the benchmark directory. Then run python -m olmocr.bench.benchmark --dir ./path/to/data to evaluate against the standard test classes. The benchmark driver will match your output files against the ground truth annotations by filename.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}