Performance Benchmarks for AllenAI/OlmOCR: Document-Level Evaluation Results

OlmOCR's benchmark suite evaluates OCR systems on 7,000 test cases across 1,400 documents, with the latest v0.4.0 release achieving an 82.4% overall score compared to 68.2% in the first release.

The allenai/olmocr repository ships a comprehensive, open-source benchmark specifically designed to test document-level OCR performance on realistic academic and historical documents. This suite comprises approximately 7,000 binary test cases spanning 1,400 PDF documents across seven distinct failure-mode categories. Understanding these performance benchmarks for allenai/olmocr helps developers compare OCR pipelines and track improvements across text extraction accuracy, table structure preservation, and mathematical formula rendering.

Benchmark Design and Methodology

The benchmark methodology is fully documented in [olmocr/bench/README.md](https://github.com/allenai/olmocr/blob/main/olmocr/bench/README.md) and emphasizes binary pass/fail evaluation to eliminate subjective scoring.

Document Categories and Test Classes

The benchmark organizes tests across seven document categories designed to stress specific OCR weaknesses:

• ArXiv Math: Scientific papers with complex mathematical notation
• Old Scans Math: Historical documents containing mathematical content
• Tables: Structured data requiring row/column alignment preservation
• Old Scans: Historical documents with degradation and noise
• Headers & Footers: Documents requiring metadata extraction from marginal regions
• Multi-Column: Complex layouts with multiple text streams
• Long Tiny Text: Documents with small font sizes and dense text blocks

Each document is evaluated against five test classes: text-presence (ensuring content appears), text-absence (ensuring hallucinations do not occur), natural reading order, table accuracy, and math formula accuracy.

Scoring and Normalization

Scores are reported as percentages with an overall weighted average across all document types. The scoring engine applies Unicode NFC normalization, hyphen/quote standardization, and case-sensitive matching (unless overridden) to ensure fair comparisons between systems. Each test is a strict binary pass/fail, making results deterministic and reproducible.

Reported Performance Benchmarks Results

The following table presents the latest published scores from the main branch, comparing OlmOCR against commercial and open-source alternatives. Scores represent percentages, with "Overall" indicating the weighted average across all document types.

Model	ArXiv	Old Scans Math	Tables	Old Scans	Headers & Footers	Multi-Column	Long Tiny Text	Base	Overall
Mistral OCR API	77.2	67.5	60.6	29.3	93.6	71.3	77.1	99.4	72.0 ± 1.1
Marker 1.10.1	83.8	66.8	72.9	33.5	86.6	80.0	85.7	99.3	76.1 ± 1.1
MinerU 2.5.4*	76.6	54.6	84.9	33.7	96.6	78.2	83.5	93.7	75.2 ± 1.1
DeepSeek-OCR	77.2	73.6	80.2	33.3	96.1	66.4	79.4	99.8	75.7 ± 1.0
Nanonets-OCR2-3B	75.4	46.1	86.8	40.9	32.1	81.9	93.0	99.6	69.5 ± 1.1
PaddleOCR-VL*	85.7	71.0	84.1	37.8	97.0	79.9	85.7	98.5	80.0 ± 1.0
Infinity-Parser 7B*	84.4	83.8	85.0	47.9	88.7	84.2	86.4	99.8	82.5 ± ?
Chandra OCR 0.1.0*	82.2	80.3	88.0	50.4	90.8	81.2	92.3	99.9	83.1 ± 0.9
OlmOCR (first release)	63.3	67.5	62.3	38.6	93.4	67.6	54.8	97.9	68.2 ± 1.1
v0.4.0 + Synth data, RLVR	83.0	82.3	84.9	47.7	96.1	83.7	81.9	99.7	82.4 ± 1.1

Rows marked with "*" indicate results reported by external model authors.

The OlmOCR v0.4.0 release demonstrates substantial improvement over the initial 68.2% baseline, achieving competitive performance with 82.4% overall accuracy through synthetic data training and RLVR (Reinforcement Learning from Visual Reasoning).

Implementation Architecture

The benchmark implementation follows a modular pipeline architecture defined in the olmocr/bench package.

Core Scoring Engine

The [olmocr/bench/benchmark.py](https://github.com/allenai/olmocr/blob/main/olmocr/bench/benchmark.py) entry point loads generated OCR output and evaluates it against ground-truth JSON annotations. This module handles the binary pass/fail logic for all five test classes and aggregates per-type scores into the final weighted average.

Conversion Pipeline

The [olmocr/bench/convert.py](https://github.com/allenai/olmocr/blob/main/olmocr/bench/convert.py) module and its runners/ sub-package provide pluggable interfaces for different OCR backends. This architecture allows researchers to benchmark custom pipelines by implementing a standard conversion interface that transforms PDFs into the Markdown/Text format required for scoring.

Metrics Tracking

For throughput analysis during large-scale evaluation, the MetricsKeeper class in olmocr/metrics.py tracks token-level processing speeds. This utility is essential for comparing not just accuracy but also computational efficiency across different OCR implementations.

Running the Benchmarks Locally

You can reproduce the published performance benchmarks for allenai/olmocr using the following end-to-end workflow. This requires Python 3.11, Playwright for math rendering verification, and the benchmark dataset from Hugging Face.

# 1️⃣ Create a fresh environment

conda create -n olmocr python=3.11 -y
conda activate olmocr

# 2️⃣ Clone the repo and install benchmark extras

git clone https://github.com/allenai/olmocr.git
cd olmocr
pip install -e .[bench]

# 3️⃣ Install the headless browser needed for math rendering

playwright install chromium

# 4️⃣ Download the benchmark dataset from Hugging Face

huggingface-cli download \
    --repo-type dataset \
    --resume-download allenai/olmOCR-bench \
    --local-dir ./olmOCR-bench

# 5️⃣ Convert PDFs using your OCR pipeline

python -m olmocr.bench.convert \
    olmocr_pipeline \
    --dir ./olmOCR-bench/bench_data

# 6️⃣ Run the scoring script

python -m olmocr.bench.benchmark --dir ./olmOCR-bench/bench_data

For distributed processing or cloud deployment, the repository provides [scripts/run_api_benchmark.sh](https://github.com/allenai/olmocr/blob/main/scripts/run_api_benchmark.sh), which packages the benchmark into a Docker image and executes it as a Beaker experiment on remote clusters.

Summary

• OlmOCR v0.4.0 achieves 82.4% overall accuracy on the document-level benchmark, representing a 14.2 percentage point improvement over the initial release.
• The benchmark evaluates 7,000 test cases across 1,400 documents in seven distinct categories including math, tables, and multi-column layouts.
• Binary pass/fail scoring with Unicode normalization ensures deterministic, reproducible results across different OCR systems.
• The complete evaluation pipeline is open-source and available in the olmocr/bench package, with the MetricsKeeper class providing additional throughput analytics.
• New OCR pipelines can be evaluated by implementing the conversion interface in olmocr.bench.convert and running the standard benchmark script.

Frequently Asked Questions

What document types does the OlmOCR benchmark evaluate?

The benchmark includes seven categories: ArXiv Math, Old Scans Math, Tables, Old Scans, Headers & Footers, Multi-Column, and Long Tiny Text. Each category targets specific OCR failure modes such as mathematical notation, historical document degradation, complex table structures, and small font sizes.

How is the overall benchmark score calculated?

The overall score is a weighted average of binary pass/fail results across approximately 7,000 individual test cases. Each test checks for exact text presence, text absence, reading order preservation, table accuracy, or math formula rendering. The scoring applies Unicode NFC normalization and hyphen standardization before performing case-sensitive matching.

Where can I find the official performance benchmark results for allenai/olmocr?

Official results are published in the repository's [olmocr/bench/README.md](https://github.com/allenai/olmocr/blob/main/olmocr/bench/README.md) file. The benchmark dataset itself is hosted on Hugging Face at allenai/olmOCR-bench, and the scoring logic is implemented in olmocr/bench/benchmark.py.

How do I add a custom OCR pipeline to the benchmark?

Implement a conversion wrapper in the olmocr.bench.convert module that transforms your OCR output into the required Markdown/Text format. Then run python -m olmocr.bench.convert your_pipeline_name --dir ./olmOCR-bench/bench_data followed by the standard benchmark command to obtain comparable scores against the published results.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}