# Core Functionalities of OlmOCR: Converting PDFs and Images to Structured Markdown with Vision-Language Models

> OlmOCR converts PDFs and images to Markdown using a Vision-Language Model. Discover its core functionalities: high-throughput processing, S3 integration, auto-rotation, and fallback extraction.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: overview
- Published: 2026-07-07

---

**OlmOCR is a high-throughput OCR pipeline that converts PDF and image documents into clean Markdown text using a Vision-Language Model, featuring distributed S3-backed processing, automatic rotation correction, and fallback extraction capabilities.**

The `allenai/olmocr` repository provides a modular document processing system designed to extract structured text from scanned documents at scale. Understanding the core functionalities of OlmOCR reveals how it leverages Vision-Language Models (VLMs) to preserve document structure—including tables, equations, and formatting—while maintaining compatibility with both local GPU inference and remote vLLM servers.

## Input Handling and Document Discovery

### WorkQueue and Distributed File Discovery

The pipeline begins with the `WorkQueue` class in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py), which manages document distribution across single machines or clusters. The `WorkQueue.populate_queue` and `WorkQueue.initialize_queue` methods (lines 27-56) parse index CSV files and group file paths into `WorkItem` objects for parallel processing.

OlmOCR accepts diverse input sources: local file paths, S3 URLs, or tarball archives. The `S3Backend` class in [`olmocr/s3_utils.py`](https://github.com/allenai/olmocr/blob/main/olmocr/s3_utils.py) handles glob expansion, ZSTD-compressed CSV management, and direct S3 I/O, enabling the pipeline to process millions of documents stored in cloud buckets without local storage constraints.

### Image Conversion and PDF Rendering

When processing non-PDF inputs, the `convert_image_to_pdf_bytes` function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 88-94) transforms PNG/JPEG files into PDF byte streams on-the-fly. For each document, the `render_pdf_to_base64png` function (implemented in [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py)) rasterizes individual pages to PNG format encoded as base64 strings.

This rendering process uses a semaphore to prevent CPU oversubscription during page rasterization. The resulting base64 images feed directly into the `build_page_query` function (lines 10-14 in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)), which prepares the visual input for the VLM.

## Vision-Language Model Inference

### Prompt Construction and Temperature Scheduling

The inference stage relies on `build_page_query` (lines 36-44 in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)) to assemble prompts by combining a fixed "no-anchoring" YAML prompt template with the base64-encoded page image. Prompt templates reside in `olmocr/prompts/*.py`, providing structured instructions that guide the model to output clean Markdown while preserving document layout.

To handle model hallucinations or repetitive generation failures, OlmOCR implements the `TEMPERATURE_BY_ATTEMPT` list (lines 84-86). This temperature scheduling system adjusts randomness across retry attempts, helping the VLM recover from generation errors without manual intervention.

### Flexible Deployment Modes

OlmOCR supports two primary inference architectures through [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). When the `--server` flag is provided, the `apost` function (lines 78-90) transmits requests to an OpenAI-compatible HTTP endpoint. For local execution, the `vllm_server_task` coroutine (lines 106-126) spawns a dedicated vLLM inference server process on the local GPU, automatically managing server lifecycle and request batching.

This dual-mode architecture allows the same codebase to operate on standalone workstations or integrate with existing Kubernetes deployments hosting vLLM clusters.

### Rotation Correction and Fallback Mechanisms

The `process_page` function (lines 84-115) implements robust error handling including automatic rotation detection. When the VLM flags a page as mis-rotated, the pipeline retries with the suggested rotation angle before exhausting attempts. If the model fails to produce valid text or the error rate exceeds configured thresholds, `make_fallback_result` (lines 33-45) initiates a fallback to the classical `pdftotext` extractor, ensuring processing continuity even for corrupted or complex documents.

## Post-Processing and Output Generation

### Dolma Document Assembly

After individual page processing, the `build_dolma_document` function (lines 200-250 in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)) aggregates page-level results into a unified document structure. This function concatenates page texts, collects token statistics, and formats output according to the Dolma JSON Lines specification—a standardized format for large-scale text datasets used in language model training.

### Markdown Export and Metrics Tracking

When invoked with the `--markdown` flag, OlmOCR utilizes `get_markdown_path` (lines 54-96) to generate parallel directory structures containing `.md` files alongside the Dolma JSON output. The `MetricsKeeper` and `WorkerTracker` classes, coordinated through the `metrics_reporter` function (lines 83-90), maintain per-worker statistics including processed tokens, page counts, and error rates, logging performance metrics periodically during batch operations.

## Distributed Processing Architecture

### S3-Backed Work Queue Management

The `WorkQueue` class abstracts storage backends through `LocalBackend` and `S3Backend` implementations, allowing seamless scaling from single-machine processing to distributed clusters. Workers pull `WorkItem` objects from the queue, process documents independently, and write results back to S3 or local filesystems without coordination bottlenecks.

### Worker Coordination and Lock Management

Each worker process in the distributed pipeline creates exclusive locks via `create_worker_lock` to prevent duplicate processing. Upon successful completion, workers write done flags using `create_done_flag`, enabling the queue to skip completed items during resume operations. This idempotent design ensures fault tolerance across machine failures or spot instance terminations in cloud environments.

## Command-Line and API Usage

### Basic CLI Conversion

Install the lightweight package and convert local documents:

```bash

# Install the lightweight package (no GPU deps)

pip install olmocr

# Convert a single PDF locally (GPU required)

olmocr ./my_workspace --markdown --pdfs my_paper.pdf

```

Connect to remote inference endpoints for distributed processing:

```bash

# Use a remote vLLM endpoint instead of a local GPU

olmocr ./my_workspace \
    --server http://my-vllm:8000/v1 \
    --model allenai/olmOCR-2-7B-1025-FP8 \
    --markdown \
    --pdfs s3://my-bucket/papers/*.pdf

```

### Programmatic Python API

Integrate OlmOCR into existing Python workflows:

```python
from olmocr.pipeline import main
import sys
import asyncio

# Simulate a CLI call – useful for custom scripts or Jupyter notebooks

sys.argv = [
    "olmocr",                     # dummy program name

    "./workspace",                # workspace directory

    "--pdfs", "sample.pdf",       # one local PDF

    "--markdown",                 # also write markdown

    "--workers", "4",             # parallel workers

]

# Run the async entry point (it will start the event-loop internally)

asyncio.run(main())

```

### Batch Processing with S3

Submit large-scale batches to S3-backed queues:

```python
from olmocr.work_queue import WorkQueue, S3Backend
import boto3
import asyncio

async def submit_batch():
    # Create a backend that points at an S3 workspace

    s3_client = boto3.client("s3")
    backend = S3Backend(s3_client, "s3://my-workspace")

    # Create the queue and populate it with 200 PDF URLs

    q = WorkQueue(backend)
    await q.populate_queue(
        work_paths=[f"s3://my-bucket/docs/{i}.pdf" for i in range(200)],
        items_per_group=10,
    )
    await q.initialize_queue()

    # Launch workers (in practice, distribute across multiple machines)

    await asyncio.gather(*[worker(args, q, i) for i in range(4)])

# asyncio.run(submit_batch())

```

## Summary

- **Multi-modal Input Processing**: OlmOCR handles PDFs, images, and tarballs through `WorkQueue` in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py), with automatic image-to-PDF conversion via `convert_image_to_pdf_bytes`.

- **VLM-Based Text Extraction**: The pipeline uses `build_page_query` and temperature scheduling to extract structured Markdown from rasterized pages, preserving tables, equations, and formatting.

- **Resilient Error Handling**: Automatic rotation correction and `pdftotext` fallback via `make_fallback_result` ensure robust processing of degraded or misaligned documents.

- **Flexible Deployment**: Supports both local vLLM servers (`vllm_server_task`) and remote OpenAI-compatible endpoints (`apost`) through unified interfaces in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py).

- **Scalable Architecture**: S3-backed work queues with worker locking and completion tracking enable fault-tolerant processing across distributed clusters.

## Frequently Asked Questions

### What makes OlmOCR different from traditional OCR tools?

Unlike traditional rule-based OCR systems, OlmOCR leverages a Vision-Language Model to understand document structure contextually, enabling accurate extraction of complex elements like mathematical equations and tables while outputting clean Markdown. The architecture specifically supports high-throughput batch processing via S3-backed distributed queues, making it suitable for processing millions of documents in cloud environments.

### Can OlmOCR process image files directly, or only PDFs?

OlmOCR handles both formats natively. When provided with PNG or JPEG files, the `convert_image_to_pdf_bytes` function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 88-94) converts images to PDF byte streams on-the-fly before rasterization, allowing the same pipeline to process scanned images, photographs of documents, and native PDFs without separate workflows.

### How does OlmOCR handle document rotation errors?

The `process_page` function implements automatic rotation detection by analyzing the VLM's output flags. When the model indicates a page is mis-rotated, the pipeline retries the request with the suggested rotation angle applied. If rotation correction fails, the system falls back to the `pdftotext` extractor via `make_fallback_result` to ensure text extraction succeeds regardless of orientation issues.

### What is the Dolma document format used by OlmOCR?

Dolma is a JSON Lines format specification designed for large-scale text datasets, implemented in `build_dolma_document` (lines 200-250 in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)). This format includes metadata fields for token counts, source attribution, and text content, making OlmOCR outputs immediately compatible with language model training pipelines and research datasets while optionally exporting parallel Markdown files for human readability.