Core Functionalities of OlmOCR: Converting PDFs and Images to Structured Markdown with Vision-Language Models

OlmOCR is a high-throughput OCR pipeline that converts PDF and image documents into clean Markdown text using a Vision-Language Model, featuring distributed S3-backed processing, automatic rotation correction, and fallback extraction capabilities.

The allenai/olmocr repository provides a modular document processing system designed to extract structured text from scanned documents at scale. Understanding the core functionalities of OlmOCR reveals how it leverages Vision-Language Models (VLMs) to preserve document structure—including tables, equations, and formatting—while maintaining compatibility with both local GPU inference and remote vLLM servers.

Input Handling and Document Discovery

WorkQueue and Distributed File Discovery

The pipeline begins with the WorkQueue class in olmocr/work_queue.py, which manages document distribution across single machines or clusters. The WorkQueue.populate_queue and WorkQueue.initialize_queue methods (lines 27-56) parse index CSV files and group file paths into WorkItem objects for parallel processing.

OlmOCR accepts diverse input sources: local file paths, S3 URLs, or tarball archives. The S3Backend class in olmocr/s3_utils.py handles glob expansion, ZSTD-compressed CSV management, and direct S3 I/O, enabling the pipeline to process millions of documents stored in cloud buckets without local storage constraints.

Image Conversion and PDF Rendering

When processing non-PDF inputs, the convert_image_to_pdf_bytes function in olmocr/pipeline.py (lines 88-94) transforms PNG/JPEG files into PDF byte streams on-the-fly. For each document, the render_pdf_to_base64png function (implemented in olmocr/data/renderpdf.py) rasterizes individual pages to PNG format encoded as base64 strings.

This rendering process uses a semaphore to prevent CPU oversubscription during page rasterization. The resulting base64 images feed directly into the build_page_query function (lines 10-14 in olmocr/pipeline.py), which prepares the visual input for the VLM.

Vision-Language Model Inference

Prompt Construction and Temperature Scheduling

The inference stage relies on build_page_query (lines 36-44 in olmocr/pipeline.py) to assemble prompts by combining a fixed "no-anchoring" YAML prompt template with the base64-encoded page image. Prompt templates reside in olmocr/prompts/*.py, providing structured instructions that guide the model to output clean Markdown while preserving document layout.

To handle model hallucinations or repetitive generation failures, OlmOCR implements the TEMPERATURE_BY_ATTEMPT list (lines 84-86). This temperature scheduling system adjusts randomness across retry attempts, helping the VLM recover from generation errors without manual intervention.

Flexible Deployment Modes

OlmOCR supports two primary inference architectures through olmocr/pipeline.py. When the --server flag is provided, the apost function (lines 78-90) transmits requests to an OpenAI-compatible HTTP endpoint. For local execution, the vllm_server_task coroutine (lines 106-126) spawns a dedicated vLLM inference server process on the local GPU, automatically managing server lifecycle and request batching.

This dual-mode architecture allows the same codebase to operate on standalone workstations or integrate with existing Kubernetes deployments hosting vLLM clusters.

Rotation Correction and Fallback Mechanisms

The process_page function (lines 84-115) implements robust error handling including automatic rotation detection. When the VLM flags a page as mis-rotated, the pipeline retries with the suggested rotation angle before exhausting attempts. If the model fails to produce valid text or the error rate exceeds configured thresholds, make_fallback_result (lines 33-45) initiates a fallback to the classical pdftotext extractor, ensuring processing continuity even for corrupted or complex documents.

Post-Processing and Output Generation

Dolma Document Assembly

After individual page processing, the build_dolma_document function (lines 200-250 in olmocr/pipeline.py) aggregates page-level results into a unified document structure. This function concatenates page texts, collects token statistics, and formats output according to the Dolma JSON Lines specification—a standardized format for large-scale text datasets used in language model training.

Markdown Export and Metrics Tracking

When invoked with the --markdown flag, OlmOCR utilizes get_markdown_path (lines 54-96) to generate parallel directory structures containing .md files alongside the Dolma JSON output. The MetricsKeeper and WorkerTracker classes, coordinated through the metrics_reporter function (lines 83-90), maintain per-worker statistics including processed tokens, page counts, and error rates, logging performance metrics periodically during batch operations.

Distributed Processing Architecture

S3-Backed Work Queue Management

The WorkQueue class abstracts storage backends through LocalBackend and S3Backend implementations, allowing seamless scaling from single-machine processing to distributed clusters. Workers pull WorkItem objects from the queue, process documents independently, and write results back to S3 or local filesystems without coordination bottlenecks.

Worker Coordination and Lock Management

Each worker process in the distributed pipeline creates exclusive locks via create_worker_lock to prevent duplicate processing. Upon successful completion, workers write done flags using create_done_flag, enabling the queue to skip completed items during resume operations. This idempotent design ensures fault tolerance across machine failures or spot instance terminations in cloud environments.

Command-Line and API Usage

Basic CLI Conversion

Install the lightweight package and convert local documents:

# Install the lightweight package (no GPU deps)

pip install olmocr

# Convert a single PDF locally (GPU required)

olmocr ./my_workspace --markdown --pdfs my_paper.pdf

Connect to remote inference endpoints for distributed processing:

# Use a remote vLLM endpoint instead of a local GPU

olmocr ./my_workspace \
    --server http://my-vllm:8000/v1 \
    --model allenai/olmOCR-2-7B-1025-FP8 \
    --markdown \
    --pdfs s3://my-bucket/papers/*.pdf

Programmatic Python API

Integrate OlmOCR into existing Python workflows:

from olmocr.pipeline import main
import sys
import asyncio

# Simulate a CLI call – useful for custom scripts or Jupyter notebooks

sys.argv = [
    "olmocr",                     # dummy program name

    "./workspace",                # workspace directory

    "--pdfs", "sample.pdf",       # one local PDF

    "--markdown",                 # also write markdown

    "--workers", "4",             # parallel workers

]

# Run the async entry point (it will start the event-loop internally)

asyncio.run(main())

Batch Processing with S3

Submit large-scale batches to S3-backed queues:

from olmocr.work_queue import WorkQueue, S3Backend
import boto3
import asyncio

async def submit_batch():
    # Create a backend that points at an S3 workspace

    s3_client = boto3.client("s3")
    backend = S3Backend(s3_client, "s3://my-workspace")

    # Create the queue and populate it with 200 PDF URLs

    q = WorkQueue(backend)
    await q.populate_queue(
        work_paths=[f"s3://my-bucket/docs/{i}.pdf" for i in range(200)],
        items_per_group=10,
    )
    await q.initialize_queue()

    # Launch workers (in practice, distribute across multiple machines)

    await asyncio.gather(*[worker(args, q, i) for i in range(4)])

# asyncio.run(submit_batch())

Summary

• Multi-modal Input Processing: OlmOCR handles PDFs, images, and tarballs through WorkQueue in olmocr/work_queue.py, with automatic image-to-PDF conversion via convert_image_to_pdf_bytes.

• VLM-Based Text Extraction: The pipeline uses build_page_query and temperature scheduling to extract structured Markdown from rasterized pages, preserving tables, equations, and formatting.

• Resilient Error Handling: Automatic rotation correction and pdftotext fallback via make_fallback_result ensure robust processing of degraded or misaligned documents.

• Flexible Deployment: Supports both local vLLM servers (vllm_server_task) and remote OpenAI-compatible endpoints (apost) through unified interfaces in olmocr/pipeline.py.

• Scalable Architecture: S3-backed work queues with worker locking and completion tracking enable fault-tolerant processing across distributed clusters.

Frequently Asked Questions

What makes OlmOCR different from traditional OCR tools?

Unlike traditional rule-based OCR systems, OlmOCR leverages a Vision-Language Model to understand document structure contextually, enabling accurate extraction of complex elements like mathematical equations and tables while outputting clean Markdown. The architecture specifically supports high-throughput batch processing via S3-backed distributed queues, making it suitable for processing millions of documents in cloud environments.

Can OlmOCR process image files directly, or only PDFs?

OlmOCR handles both formats natively. When provided with PNG or JPEG files, the convert_image_to_pdf_bytes function in olmocr/pipeline.py (lines 88-94) converts images to PDF byte streams on-the-fly before rasterization, allowing the same pipeline to process scanned images, photographs of documents, and native PDFs without separate workflows.

How does OlmOCR handle document rotation errors?

The process_page function implements automatic rotation detection by analyzing the VLM's output flags. When the model indicates a page is mis-rotated, the pipeline retries the request with the suggested rotation angle applied. If rotation correction fails, the system falls back to the pdftotext extractor via make_fallback_result to ensure text extraction succeeds regardless of orientation issues.

What is the Dolma document format used by OlmOCR?

Dolma is a JSON Lines format specification designed for large-scale text datasets, implemented in build_dolma_document (lines 200-250 in olmocr/pipeline.py). This format includes metadata fields for token counts, source attribution, and text content, making OlmOCR outputs immediately compatible with language model training pipelines and research datasets while optionally exporting parallel Markdown files for human readability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}