olmOCR Performance Benchmarks: How the 82.4% Overall Score Is Calculated

The olmOCR v0.4.0 model achieves an 82.4% ± 1.1% overall score on the olmOCR-Bench evaluation suite, outperforming Marker 1.10.1 (76.1%) and PaddleOCR-VL (80.0%) through strict unit-test-style validation across eight document categories.

The allenai/olmocr repository provides a reproducible benchmark suite called olmOCR-Bench that quantifies OCR pipeline accuracy using automated fact-checking rather than fuzzy text matching. Understanding these performance benchmarks for olmOCR is essential for comparing document extraction tools and validating improvements across model releases.

What Is the olmOCR-Bench Suite?

The olmOCR-Bench suite provides a quantitative, reproducible way to evaluate the OCR quality of any pipeline that outputs plain-text or Markdown. Unlike traditional benchmarks that rely on edit distance or similarity metrics, olmOCR-Bench uses strict presence/absence checks against ground-truth facts.

Directory Structure and Test Files

The benchmark reads a directory containing three components:

• A pdfs/ folder with the source PDFs
• One or more candidate sub-folders that hold the OCR outputs as per-page Markdown files (e.g., doc1_pg1_repeat1.md)
• A set of .jsonl files that describe unit-test-style facts about each page, covering text presence, header-footer removal, table accuracy, and math-formula accuracy

Scoring Methodology

The core driver in olmocr/bench/benchmark.py loads all tests from the .jsonl files, matches each test to the corresponding Markdown files, and runs the test logic concurrently. The evaluate_candidate function (lines 33-44) handles the evaluation loop, while the main function (lines 53-90) aggregates scores per-JSONL file and computes confidence intervals via bootstrap sampling. The final overall score represents the average of per-JSONL pass rates.

Current Performance Benchmarks and Results

The repository’s top-level README.md (lines 70-83) contains the latest published results. For the current v0.4.0 release, the overall score is 82.4% ± 1.1%.

System	ArXiv	Old scans (math)	Tables	Old scans	Headers & footers	Multi column	Long tiny text	Base	Overall
olmOCR v0.4.0	83.0	82.3	84.9	47.7	96.1	83.7	81.9	99.7	82.4 ± 1.1
Marker 1.10.1	83.8	66.8	72.9	33.5	86.6	80.0	85.7	99.3	76.1 ± 1.1
PaddleOCR-VL*	85.7	71.0	84.1	37.8	97.0	79.9	85.7	98.5	80.0 ± 1.0

Scores for earlier releases (v0.1-v0.3) show a steady improvement of roughly 4 percentage points from the first release (68.2% ± 1.1%) to v0.4.0. The "Overall" column reflects the average of the per-category pass rates and is accompanied by a 95% confidence interval.

Benchmark Methodology and Scoring

The benchmark deliberately avoids soft metrics like edit distance, focusing instead on strict unit-test-style validation. This approach makes scores directly comparable across different OCR tools.

Confidence Interval Calculation

Confidence intervals are computed by bootstrap sampling over the per-JSONL scores. This statistical method ensures that the reported margins (e.g., ± 1.1%) accurately reflect the variance across the diverse document set, which includes ArXiv papers, old scanned documents with math, tables, multi-column layouts, and tiny text.

Interpretation of Scores

Higher numbers indicate that the OCR pipeline correctly reproduces all unit-test facts for a given document category. For example, the 96.1% score in "Headers & footers" means olmOCR v0.4.0 successfully removes header and footer text in nearly all test cases, while the 47.7% score on "Old scans" reflects the ongoing challenge of degraded historical documents.

How to Run the olmOCR Benchmark

You can reproduce these performance benchmarks for olmOCR on your local machine using the benchmark driver.

Installation and Setup

# Install the benchmark dependencies

pip install "olmocr[bench]"

# Download the benchmark dataset (requires huggingface-cli)

huggingface-cli download --repo-type dataset \
  allenai/olmOCR-bench --local-dir ./olmOCR-bench

Running the Evaluation

# Convert your PDFs with the pipeline you want to evaluate

python -m olmocr.pipeline ./workspace \
  --markdown --pdfs ./olmOCR-bench/pdfs/**/*.pdf

# Move the pipeline output into the benchmark format

python -m olmocr.bench.scripts.workspace_to_bench \
  ./workspace ./olmOCR-bench/bench_data/olmocr \
  --bench-path ./olmOCR-bench

# Run the benchmark

python -m olmocr.bench.benchmark \
  --dir ./olmOCR-bench/bench_data

Add --test_report results.html to generate a full HTML report with detailed breakdowns.

Programmatic Usage

You can also extract scores programmatically using the evaluate_candidate function:

from olmocr.bench.benchmark import evaluate_candidate

# candidate_folder = path to the folder that contains your MD output

overall_score, *_ = evaluate_candidate(candidate_folder, all_tests, pdf_basenames)
print(f"olmOCR benchmark score: {overall_score * 100:.1f}%")

Summary

• olmOCR v0.4.0 achieves an 82.4% ± 1.1% overall score on the olmOCR-Bench suite, leading Marker (76.1%) and PaddleOCR-VL (80.0%).
• The benchmark uses strict unit-test-style validation via .jsonl fact files rather than fuzzy text matching.
• Scores are aggregated in olmocr/bench/benchmark.py using bootstrap sampling to compute 95% confidence intervals.
• The suite covers eight document categories, with olmOCR showing particular strength in header/footer removal (96.1%) and table extraction (84.9%).
• You can reproduce results locally using python -m olmocr.bench.benchmark after converting PDFs to the required Markdown format.

Frequently Asked Questions

What does the 82.4% overall score represent?

The 82.4% score represents the average pass rate across all unit-test validations in the olmOCR-Bench suite. According to the source code in olmocr/bench/benchmark.py, the final aggregation logic computes this by averaging the per-JSONL pass rates after bootstrap sampling. A score of 82.4% means that, on average, the pipeline correctly handles 82.4% of the strict fact-checking tests across diverse document types including ArXiv papers, scanned books, and tables.

How does olmOCR-Bench differ from other OCR benchmarks?

Unlike traditional benchmarks that use edit distance or BLEU scores, olmOCR-Bench uses strict presence/absence checks defined in .jsonl files. As implemented in allenai/olmocr, the benchmark validates whether specific text, math formulas, or table structures are exactly present in the output Markdown. This approach eliminates fuzzy matching and ensures that scores reflect genuine accuracy rather than approximate similarity.

How can I reproduce the benchmark results on my own PDFs?

To reproduce the performance benchmarks for olmOCR on your documents, install the benchmark dependencies with pip install "olmocr[bench]", then use the olmocr.bench.benchmark module as shown in the code examples above. The evaluate_candidate function in olmocr/bench/benchmark.py handles the core logic for matching your Markdown outputs against the ground-truth .jsonl files and computing the final score with confidence intervals.

What confidence intervals are used and why?

The benchmark reports 95% confidence intervals computed via bootstrap sampling over per-JSONL scores. The implementation in benchmark.py (lines 53-90) resamples the test results to estimate the variance, producing intervals like ± 1.1%. This statistical method ensures that the reported scores account for the diversity of the document set and provide a reliable estimate of true performance across different document categories.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}