# Structure and Categories of the olmOCR-Bench Benchmark Suite: A Complete Guide

> Explore the olmOCR-Bench benchmark suite's structure and categories. Discover seven document types and five test classes for evaluating OCR systems. Learn more at allenai/olmocr.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: deep-dive
- Published: 2026-07-06

---

**The olmOCR-Bench benchmark suite evaluates document-level OCR systems through seven distinct document types and five rigorous test classes, using JSON annotations stored in `olmocr/bench/sample_data/` and executed via [`benchmark.py`](https://github.com/allenai/olmocr/blob/main/benchmark.py).**

The olmOCR-Bench benchmark suite is the standardized evaluation framework for the allenai/olmocr repository, designed to rigorously test how well OCR pipelines handle complex single-page PDF documents. It combines curated document corpora with automated test generation to measure accuracy across math formulas, tables, reading order preservation, and text extraction fidelity.

## Document Categories: The Seven Source Types

The **olmOCR-Bench** suite organizes its corpus into seven distinct **document types**, each targeting known failure modes of OCR systems. According to the repository README (lines 220-236), these categories are:

- **ArXiv Math (AR)**: Recent mathematics papers sourced from arXiv. These single-TeX source PDFs challenge OCR systems with complex LaTeX formulas. Candidate pages are identified via OCR, matched back to source, validated with KaTeX, and manually verified.

- **Old Scans Math (OSM)**: Public-domain scanned mathematics textbooks from the Internet Archive. Formula-containing pages are identified by OCR, with each formula manually annotated to create ground truth.

- **Tables (TA)**: PDFs containing tabular data drawn from the same internal crawl used for olmOCR-Mix. Gemini-Flash-2.0 selects random cells, generates relational queries, and manual review ensures correctness.

- **Old Scans (OS)**: Historical letters and typewritten documents with existing human transcriptions from the Library of Congress. Scripts produce natural-reading-order text pairs, with header and footer cases added and human-reviewed.

- **Headers Footers (HF)**: Regions identified by DocLayout-YOLO on the internal crawl. Visual masking combined with Gemini-Flash-2.0 extracts text; humans prune false positives and set search-area limits.

- **Multi Column (MC)**: Multi-column page layouts sampled visually from the internal PDF pool. Claude-Sonnet renders pages to HTML, extracts adjacent text segments, and hand-checking validates the reading order.

- **Long Tiny Text (LTT)**: Dense, small-print pages such as dictionary entries and reference lists. Gemini-Flash-2.0 creates test cases followed by manual verification.

These **document types** are stored in `olmocr/bench/sample_data/pdfs/`, with corresponding test specifications in `olmocr/bench/sample_data/olmocr_pipeline/` and a master index in `olmocr/bench/sample_data/dataset.jsonl`.

## Test Classes: Evaluation Methodology

Each benchmark item falls into one of five **test classes** that dictate how OCR output is judged, as defined in the README starting at line 53:

- **Text Presence**: Verifies that a short snippet (1-3 sentences) appears somewhere in the output. Uses soft/fuzzy matching and supports optional "first N/last N characters" constraints.

- **Text Absence**: Ensures that a given string does not appear in the output, critical for detecting header/footer noise injection. Applies the same fuzzy matching rules as Text Presence.

- **Natural Reading Order**: Confirms that two blocks of text appear in the correct relative sequence. Allows swapping of whole sections when order is non-critical.

- **Table Accuracy**: Checks that a specific cell exists and that its neighbors satisfy specified constraints. Supports Markdown tables, though full scoring requires HTML tables to properly validate rowspan and colspan attributes.

- **Math Formula Accuracy**: Renders a LaTeX expression with KaTeX and verifies that the OCR output contains the same symbolic sequence (order-insensitive). Equations must be wrapped in `$...$`, `$$...$$`, `\( ... \)`, or `\[ ... \]` delimiters.

The evaluation logic for these classes is implemented in [`olmocr/bench/benchmark.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/benchmark.py), which normalizes Unicode and performs the specific checks for each test type.

## olmOCR-Bench Architecture and Key Components

The **structure of the benchmark suite** comprises several interconnected components:

### Data Storage and Layout

The dataset resides in `olmocr/bench/sample_data/`. PDF pages live in `sample_data/pdfs/`, while JSON annotation files containing the test specifications are located in `sample_data/olmocr_pipeline/`. The master file `sample_data/dataset.jsonl` indexes all test cases across the seven document categories.

### Conversion Pipeline

[`olmocr/bench/convert.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/convert.py) transforms raw PDFs into the markdown format required by the benchmark. For users running the full olmOCR pipeline, [`olmocr/bench/scripts/workspace_to_bench.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/scripts/workspace_to_bench.py) adapts the pipeline workspace output to the benchmark layout.

### Execution Driver

[`olmocr/bench/benchmark.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/benchmark.py) serves as the core evaluation engine. It loads JSON annotations, runs the selected OCR tool on each PDF, normalizes the output, and evaluates every test case according to its designated class.

### Reporting and Analysis

[`olmocr/bench/report.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/report.py) aggregates per-category scores and produces consumable HTML and LaTeX summaries. For interactive debugging, [`olmocr/bench/review_app.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/review_app.py) launches a Flask-based web UI on a specified port to inspect individual questions and expected answers.

### OCR Runner Scripts

The `olmocr/bench/runners/` directory contains ready-to-use scripts (e.g., [`run_mistral.py`](https://github.com/allenai/olmocr/blob/main/run_mistral.py), [`run_marker.py`](https://github.com/allenai/olmocr/blob/main/run_marker.py), [`run_paddleocr.py`](https://github.com/allenai/olmocr/blob/main/run_paddleocr.py)) demonstrating how to benchmark specific OCR backends against the suite.

## Running the olmOCR-Bench Benchmark

Execute the following workflow to evaluate an OCR system against the benchmark suite.

### Installation and Setup

Install the benchmark dependencies and Playwright for KaTeX rendering:

```bash
conda create -n olmocr python=3.11
conda activate olmocr
git clone https://github.com/allenai/olmocr.git
cd olmocr
pip install -e .[bench]
playwright install chromium

```

### Downloading the Dataset

Retrieve the benchmark corpus from Hugging Face:

```bash
huggingface-cli download \
    --repo-type dataset \
    --resume-download allenai/olmOCR-bench \
    --local-dir ./olmOCR-bench

```

### Converting Documents

Convert existing PDFs to the required markdown format using the generic converter:

```bash
python -m olmocr.bench.convert olmocr_pipeline \
    --dir ./olmOCR-bench/bench_data

```

Alternatively, process raw PDFs through the main pipeline and adapt the workspace:

```bash
python -m olmocr.pipeline ./localworkspace \
    --markdown \
    --pdfs ./olmOCR-bench/bench_data/pdfs/**/*.pdf

python -m olmocr.bench.scripts.workspace_to_bench \
    localworkspace/ \
    olmOCR-bench/bench_data/olmocr \
    --bench-path ./olmOCR-bench/

```

### Executing the Benchmark

Run the evaluation driver to compute scores across all categories:

```bash
python -m olmocr.bench.benchmark \
    --dir ./olmOCR-bench/bench_data

```

The script prints per-category pass percentages and an overall aggregate score.

### Interactive Review

Launch the review application to inspect specific test cases:

```bash
python -m olmocr.bench.review_app \
    --port 5000 \
    --debug ./olmOCR-bench/bench_data/multi_column.jsonl \
    --force

```

Access the interface at `http://localhost:5000` to explore questions and ground truth annotations.

## Summary

The **olmOCR-Bench benchmark suite** provides a comprehensive framework for evaluating document OCR systems through its dual-categorization approach:

- **Seven document types** (AR, OSM, TA, OS, HF, MC, LTT) targeting specific PDF complexity domains
- **Five test classes** (Text Presence, Text Absence, Natural Reading Order, Table Accuracy, Math Formula Accuracy) providing rigorous evaluation criteria
- **Modular architecture** with clear separation between data storage (`sample_data/`), conversion tools ([`convert.py`](https://github.com/allenai/olmocr/blob/main/convert.py)), execution ([`benchmark.py`](https://github.com/allenai/olmocr/blob/main/benchmark.py)), and reporting ([`report.py`](https://github.com/allenai/olmocr/blob/main/report.py))
- **Practical tooling** including runner scripts for major OCR engines and a web-based review application

## Frequently Asked Questions

### What file formats does olmOCR-Bench require?

The benchmark requires PDF documents for input and markdown files for OCR output evaluation. Annotations are stored as JSON Lines (`.jsonl`) files in `olmocr/bench/sample_data/olmocr_pipeline/`, with a master index at `sample_data/dataset.jsonl`. The conversion pipeline in [`olmocr/bench/convert.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/convert.py) handles the transformation from raw PDF to the required markdown format.

### How does the benchmark handle math formula verification?

The **Math Formula Accuracy** test class uses KaTeX to render LaTeX expressions from the ground truth and compares them symbolically against the OCR output. According to the source code, equations must be delimited by `$...$`, `$$...$$`, `\( ... \)`, or `\[ ... \]`. The comparison is order-insensitive for symbols but requires complete symbolic sequence matching, implemented in the evaluation logic of [`olmocr/bench/benchmark.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/benchmark.py).

### Can I benchmark custom OCR pipelines not included in the runners?

Yes. Use [`olmocr/bench/convert.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/convert.py) to transform your OCR output into the benchmark's expected markdown format, or manually place markdown files (`.md`) alongside the JSON annotation files in the benchmark directory. Then run `python -m olmocr.bench.benchmark --dir ./path/to/data` to evaluate against the standard test classes. The benchmark driver will match your output files against the ground truth annotations by filename.