What is allenai/olmocr? A VLM-Powered PDF OCR Toolkit Explained

allenai/olmocr is an open-source toolkit that converts PDFs and image-based documents into clean, structured Markdown or plain text using Vision-Language Models (VLMs), preserving tables, equations, and multi-column layouts while outputting Dolma-compatible JSON-L documents.

Developed by the Allen Institute for AI, this repository provides a high-throughput pipeline designed for both single-GPU workstations and distributed cloud environments. It renders PDF pages as images, processes them through a vLLM server running models like allenai/olmOCR-2-7B-1025-FP8, and assembles the results into machine-readable documents suitable for downstream NLP tasks.

Core Architecture and Components

The repository organizes functionality into modular Python packages that handle everything from job distribution to response parsing.

CLI Entry Point and Pipeline Orchestration

The olmocr command-line interface serves as the primary entry point, exposed through olmocr/__main__.py. This module parses arguments and launches the processing pipeline defined in pipeline.py, which coordinates the entire workflow including page-wise processing, retry logic, and document assembly.

Work Queue and Distributed Processing

For scaling across multiple nodes, work_queue.py abstracts both local filesystem and S3-backed job distribution. It groups PDFs into work items based on the --pages_per_group parameter, enabling efficient multi-node processing on Beaker clusters or AWS infrastructure.

PDF Rendering and Image Processing

The renderpdf.py module converts each PDF page into a base64-encoded PNG image suitable for VLM consumption. Supporting utilities in image_utils.py handle format detection and conversion between JPEG/PNG and PDF byte streams, ensuring compatibility with diverse input sources.

Prompt Engineering and Response Parsing

The system uses carefully crafted prompts defined in prompts/prompts.py and prompts/anchor.py to instruct the VLM. These modules build the system prompt and parse the model's YAML-formatted front-matter responses into typed PageResponse objects, extracting structured text while preserving document semantics.

Filtering and Quality Control

Before processing, filter/filter.py applies language detection and spam filters to exclude low-quality PDFs, forms, or non-English documents. This pre-filtering step saves compute resources and improves output quality.

Metrics and Monitoring

The metrics.py module tracks token usage, request rates, and worker statistics throughout the pipeline, providing observability for large-scale batch jobs.

Document Viewer

For inspecting results, viewer/dolmaviewer.py provides a minimal HTML interface to browse the generated Dolma documents locally without requiring external tools.

End-to-End Data Flow

The pipeline follows a robust eight-step process to handle document conversion:

1. Workspace Initialization – Users specify a workspace directory (local path or s3:// URI) that serves as the hub for input and output.

2. Work Item Creation – The WorkQueue groups PDFs or tarballs into discrete work items based on the --pages_per_group configuration.

3. Worker Initialization – Each worker() process pulls work items from the queue and invokes process_pdf() or process_tarball().

4. Page Rendering – The process_page() function calls render_pdf_to_base64png to generate image data, then constructs a VLM query via build_page_query().

5. VLM Inference – The system sends requests via apost() to the vLLM server, handling both local and remote OpenAI-compatible endpoints.

6. Retry and Rotation Handling – If the model detects invalid page rotation, the pipeline automatically corrects the angle and retries. Other failures trigger exponential back-off with up to --max_page_retries attempts.

7. Fallback Extraction – When VLM processing fails completely, the system executes make_fallback_result() using pdftotext to ensure some text extraction occurs.

8. Document Assembly – Finally, build_dolma_document() concatenates per-page results, attaches metadata (model version, token counts), and writes Dolma JSON-L records to the workspace.

Installation and Usage Patterns

Local GPU Processing

For maximum throughput on dedicated hardware, install the GPU-enabled package and run locally:

# Create environment with Python 3.11

conda create -n olmocr python=3.11
conda activate olmocr

# Install with CUDA support

pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128

# Convert PDF to Markdown

curl -o sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf
olmocr ./workspace --markdown --pdfs sample.pdf

The CLI automatically starts a local vLLM server (vllm_server_task) and writes Markdown output to ./workspace/markdown/.

Remote Inference (Lightweight)

For environments without GPUs, use an external vLLM endpoint:

# Install lightweight variant

pip install olmocr

# Point to remote server

olmocr ./workspace \
  --server http://my-vllm-host:8000/v1 \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --markdown \
  --pdfs *.pdf

This configuration directs pipeline.py to call apost() against the specified endpoint, bypassing local model loading entirely.

Programmatic Python API

Integrate the pipeline directly into Python applications:

from olmocr.pipeline import main as olmocr_main
import sys
import asyncio

# Configure arguments programmatically

sys.argv = [
    "olmocr",
    "./workspace",
    "--pdfs", "sample.pdf",
    "--markdown",
    "--workers", "4",
]

# Execute the async pipeline

asyncio.run(olmocr_main())

This approach feeds command-line arguments to main() and executes the async entry point within your existing event loop.

Docker Deployment

The repository includes two container configurations for reproducible deployments:

• Dockerfile – Base image requiring external model endpoint
• Dockerfile.with-model – Self-contained image including the VLM weights

These images support both single-container local processing and scaled Beaker cluster deployments.

Benchmarking and Validation

The olmocr/bench/ directory contains olmOCR-Bench, a comprehensive evaluation suite featuring over 7,000 test cases across 1,400 documents. This benchmark validates OCR quality across categories including tables, mathematical equations, and multi-column layouts.

Run the full evaluation with:

pip install "olmocr[bench]"
python -m olmocr.bench.tests

The suite generates scores comparable to those reported in the repository README, enabling quantitative comparison against alternative OCR solutions.

Summary

• allenai/olmocr provides a production-ready pipeline for converting PDFs to structured text using Vision-Language Models.
• The architecture separates concerns across pipeline.py, work_queue.py, and specialized modules for rendering, filtering, and metrics.
• Flexible deployment options support local GPUs, remote vLLM endpoints, and distributed S3-backed processing.
• Robust error handling includes automatic rotation correction, exponential back-off retries, and pdftotext fallback.
• Complete Dockerization and benchmarking tools ensure reproducible results at scale.

Frequently Asked Questions

What file formats does allenai/olmocr support?

The toolkit primarily processes PDF documents, including scanned image-based PDFs. It also handles tarballs containing multiple PDFs and converts individual image files (JPEG, PNG) to PDF format internally using utilities in image_utils.py before processing.

How does allenai/olmocr handle pages that fail OCR processing?

The system implements a multi-layer retry mechanism defined in pipeline.py. If the VLM returns an invalid rotation error, the page is automatically re-rendered with corrected angles. For other failures, the system applies exponential back-off with configurable --max_page_retries. As a final fallback, it executes make_fallback_result() using the system's pdftotext command to extract raw text.

Can I use allenai/olmocr without a local GPU?

Yes. While the GPU-enabled installation (pip install "olmocr[gpu]") provides optimal throughput for local processing, you can run the lightweight version (pip install olmocr) and point to any OpenAI-compatible API endpoint using the --server flag. This allows CPU-only machines to leverage remote VLM inference.

What is a Dolma-compatible document?

Dolma is a document format standard used by the Allen Institute for AI for training large language models. The build_dolma_document() function in pipeline.py assembles OCR results into JSON-L records containing the extracted text, metadata (token counts, model version), and provenance information, making the output immediately usable for ML training pipelines.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}