# What is allenai/olmocr? A VLM-Powered PDF OCR Toolkit Explained

> Discover allenai/olmocr an open-source toolkit that transforms PDFs into clean structured Markdown or text using VLMs Preserves tables equations and layouts outputting Dolma-compatible JSON-L

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: getting-started
- Published: 2026-07-07

---

**allenai/olmocr is an open-source toolkit that converts PDFs and image-based documents into clean, structured Markdown or plain text using Vision-Language Models (VLMs), preserving tables, equations, and multi-column layouts while outputting Dolma-compatible JSON-L documents.**

Developed by the Allen Institute for AI, this repository provides a high-throughput pipeline designed for both single-GPU workstations and distributed cloud environments. It renders PDF pages as images, processes them through a vLLM server running models like `allenai/olmOCR-2-7B-1025-FP8`, and assembles the results into machine-readable documents suitable for downstream NLP tasks.

## Core Architecture and Components

The repository organizes functionality into modular Python packages that handle everything from job distribution to response parsing.

### CLI Entry Point and Pipeline Orchestration

The **`olmocr`** command-line interface serves as the primary entry point, exposed through [`olmocr/__main__.py`](https://github.com/allenai/olmocr/blob/main/olmocr/__main__.py). This module parses arguments and launches the processing pipeline defined in **[`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)**, which coordinates the entire workflow including page-wise processing, retry logic, and document assembly.

### Work Queue and Distributed Processing

For scaling across multiple nodes, **[`work_queue.py`](https://github.com/allenai/olmocr/blob/main/work_queue.py)** abstracts both local filesystem and S3-backed job distribution. It groups PDFs into work items based on the `--pages_per_group` parameter, enabling efficient multi-node processing on Beaker clusters or AWS infrastructure.

### PDF Rendering and Image Processing

The **[`renderpdf.py`](https://github.com/allenai/olmocr/blob/main/renderpdf.py)** module converts each PDF page into a base64-encoded PNG image suitable for VLM consumption. Supporting utilities in **[`image_utils.py`](https://github.com/allenai/olmocr/blob/main/image_utils.py)** handle format detection and conversion between JPEG/PNG and PDF byte streams, ensuring compatibility with diverse input sources.

### Prompt Engineering and Response Parsing

The system uses carefully crafted prompts defined in **[`prompts/prompts.py`](https://github.com/allenai/olmocr/blob/main/prompts/prompts.py)** and **[`prompts/anchor.py`](https://github.com/allenai/olmocr/blob/main/prompts/anchor.py)** to instruct the VLM. These modules build the system prompt and parse the model's YAML-formatted front-matter responses into typed `PageResponse` objects, extracting structured text while preserving document semantics.

### Filtering and Quality Control

Before processing, **[`filter/filter.py`](https://github.com/allenai/olmocr/blob/main/filter/filter.py)** applies language detection and spam filters to exclude low-quality PDFs, forms, or non-English documents. This pre-filtering step saves compute resources and improves output quality.

### Metrics and Monitoring

The **[`metrics.py`](https://github.com/allenai/olmocr/blob/main/metrics.py)** module tracks token usage, request rates, and worker statistics throughout the pipeline, providing observability for large-scale batch jobs.

### Document Viewer

For inspecting results, **[`viewer/dolmaviewer.py`](https://github.com/allenai/olmocr/blob/main/viewer/dolmaviewer.py)** provides a minimal HTML interface to browse the generated Dolma documents locally without requiring external tools.

## End-to-End Data Flow

The pipeline follows a robust eight-step process to handle document conversion:

1. **Workspace Initialization** – Users specify a workspace directory (local path or `s3://` URI) that serves as the hub for input and output.

2. **Work Item Creation** – The `WorkQueue` groups PDFs or tarballs into discrete work items based on the `--pages_per_group` configuration.

3. **Worker Initialization** – Each `worker()` process pulls work items from the queue and invokes `process_pdf()` or `process_tarball()`.

4. **Page Rendering** – The `process_page()` function calls `render_pdf_to_base64png` to generate image data, then constructs a VLM query via `build_page_query()`.

5. **VLM Inference** – The system sends requests via `apost()` to the vLLM server, handling both local and remote OpenAI-compatible endpoints.

6. **Retry and Rotation Handling** – If the model detects invalid page rotation, the pipeline automatically corrects the angle and retries. Other failures trigger exponential back-off with up to `--max_page_retries` attempts.

7. **Fallback Extraction** – When VLM processing fails completely, the system executes `make_fallback_result()` using `pdftotext` to ensure some text extraction occurs.

8. **Document Assembly** – Finally, `build_dolma_document()` concatenates per-page results, attaches metadata (model version, token counts), and writes Dolma JSON-L records to the workspace.

## Installation and Usage Patterns

### Local GPU Processing

For maximum throughput on dedicated hardware, install the GPU-enabled package and run locally:

```bash

# Create environment with Python 3.11

conda create -n olmocr python=3.11
conda activate olmocr

# Install with CUDA support

pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128

# Convert PDF to Markdown

curl -o sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf
olmocr ./workspace --markdown --pdfs sample.pdf

```

The CLI automatically starts a local vLLM server (`vllm_server_task`) and writes Markdown output to `./workspace/markdown/`.

### Remote Inference (Lightweight)

For environments without GPUs, use an external vLLM endpoint:

```bash

# Install lightweight variant

pip install olmocr

# Point to remote server

olmocr ./workspace \
  --server http://my-vllm-host:8000/v1 \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --markdown \
  --pdfs *.pdf

```

This configuration directs [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) to call `apost()` against the specified endpoint, bypassing local model loading entirely.

### Programmatic Python API

Integrate the pipeline directly into Python applications:

```python
from olmocr.pipeline import main as olmocr_main
import sys
import asyncio

# Configure arguments programmatically

sys.argv = [
    "olmocr",
    "./workspace",
    "--pdfs", "sample.pdf",
    "--markdown",
    "--workers", "4",
]

# Execute the async pipeline

asyncio.run(olmocr_main())

```

This approach feeds command-line arguments to `main()` and executes the async entry point within your existing event loop.

### Docker Deployment

The repository includes two container configurations for reproducible deployments:

- **`Dockerfile`** – Base image requiring external model endpoint
- **`Dockerfile.with-model`** – Self-contained image including the VLM weights

These images support both single-container local processing and scaled Beaker cluster deployments.

## Benchmarking and Validation

The **`olmocr/bench/`** directory contains **olmOCR-Bench**, a comprehensive evaluation suite featuring over 7,000 test cases across 1,400 documents. This benchmark validates OCR quality across categories including tables, mathematical equations, and multi-column layouts.

Run the full evaluation with:

```bash
pip install "olmocr[bench]"
python -m olmocr.bench.tests

```

The suite generates scores comparable to those reported in the repository README, enabling quantitative comparison against alternative OCR solutions.

## Summary

- **allenai/olmocr** provides a production-ready pipeline for converting PDFs to structured text using Vision-Language Models.
- The architecture separates concerns across [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py), [`work_queue.py`](https://github.com/allenai/olmocr/blob/main/work_queue.py), and specialized modules for rendering, filtering, and metrics.
- Flexible deployment options support local GPUs, remote vLLM endpoints, and distributed S3-backed processing.
- Robust error handling includes automatic rotation correction, exponential back-off retries, and `pdftotext` fallback.
- Complete Dockerization and benchmarking tools ensure reproducible results at scale.

## Frequently Asked Questions

### What file formats does allenai/olmocr support?

The toolkit primarily processes **PDF documents**, including scanned image-based PDFs. It also handles tarballs containing multiple PDFs and converts individual image files (JPEG, PNG) to PDF format internally using utilities in [`image_utils.py`](https://github.com/allenai/olmocr/blob/main/image_utils.py) before processing.

### How does allenai/olmocr handle pages that fail OCR processing?

The system implements a multi-layer retry mechanism defined in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py). If the VLM returns an invalid rotation error, the page is automatically re-rendered with corrected angles. For other failures, the system applies exponential back-off with configurable `--max_page_retries`. As a final fallback, it executes `make_fallback_result()` using the system's `pdftotext` command to extract raw text.

### Can I use allenai/olmocr without a local GPU?

Yes. While the GPU-enabled installation (`pip install "olmocr[gpu]"`) provides optimal throughput for local processing, you can run the lightweight version (`pip install olmocr`) and point to any OpenAI-compatible API endpoint using the `--server` flag. This allows CPU-only machines to leverage remote VLM inference.

### What is a Dolma-compatible document?

**Dolma** is a document format standard used by the Allen Institute for AI for training large language models. The `build_dolma_document()` function in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) assembles OCR results into JSON-L records containing the extracted text, metadata (token counts, model version), and provenance information, making the output immediately usable for ML training pipelines.