How Does OlmOCR Process OCR Tasks: An Eight-Stage Technical Pipeline

OlmOCR converts PDFs into structured Dolma documents by rendering each page as a base64-encoded PNG image and processing it through a vision-capable large language model, featuring automatic rotation correction, intelligent retry logic, and a pdftotext fallback mechanism.

The AllenAI OlmOCR repository implements a sophisticated async pipeline that transforms static PDF documents into machine-readable structured data. Understanding how OlmOCR processes OCR tasks requires examining its eight-stage architecture, which orchestrates PDF ingestion, vision-language model inference, and robust error recovery. The system outputs standardized Dolma JSON-L documents with rich per-page metadata stored in document attributes.

The Eight-Stage OCR Pipeline

Stage 1: PDF Loading and Ingestion

The pipeline begins in olmocr/pipeline.py where the process_pdf function (line 75) handles document acquisition. The system reads input from S3 buckets, local disk, or tarballs, converting JPEG or PNG images to PDF format when necessary. This stage establishes the async processing context and prepares the temporary workspace for downstream operations.

Stage 2: Page-by-Page Rendering

Within build_page_query (line 6), each page renders to a base64-encoded PNG with a configurable maximum longest dimension. The implementation uses a bounded async semaphore to prevent resource exhaustion, ensuring the host never spawns excessive concurrent workers. This rendering transforms document pages into the image format required by the vision-capable LLM.

Stage 3: LLM Request Construction and Inference

The try_single_page function (line 61) constructs the inference request sent to the chat completions endpoint. The payload includes the PNG image data alongside the fixed YAML prompt generated by build_no_anchoring_v4_yaml_prompt from olmocr/prompts.py. The system implements adaptive retry logic that increases temperature on each retry to break looping outputs, maximizing successful extraction rates.

Stage 4: Response Parsing and Front Matter Extraction

After receiving the model's markdown response, the FrontMatterParser (defined in olmocr/train/front_matter.py) extracts structured metadata at line 110 of the pipeline. This component parses the markdown front matter containing language detection, rotation flags, table presence, and diagram indicators into a typed PageResponse object, enabling programmatic access to page-level attributes.

Stage 5: Rotation Correction and Retry Logic

The process_page function (line 84) handles geometric corrections when the model indicates is_rotation_valid=False. The pipeline tracks cumulative_rotation across sequential attempts, rotating the page image and retrying inference up to the configurable args.max_page_retries limit. Non-rotation errors trigger parallel retry attempts once the VLLM request queue empties, optimizing throughput during error recovery.

Stage 6: OCR Fallback Mechanism

When the LLM exhausts all retry attempts, make_fallback_result (line 33) executes a pure-OCR fallback using pdftotext. This mechanism ensures text recovery even when the vision model fails to produce valid output, guaranteeing document accessibility regardless of image quality or layout complexity.

Stage 7: Dolma Document Assembly

The build_dolma_document function (line 202) aggregates all PageResult objects into a single structured document. The concatenated natural text forms the document body, while per-page metadata—including language codes, rotation angles, and structural flags—populates the attributes dictionary. This format aligns with the Dolma corpus specification for downstream NLP training workflows.

Stage 8: Output Generation and Persistence

Finally, the worker function (line 98) persists the completed document to the workspace or S3 as JSON-L. When the --markdown flag is enabled, the pipeline also emits a plain-text .md version alongside the structured output, accommodating both human review and machine processing requirements.

Distributed Processing Architecture

Beyond single-document processing, the WorkQueue class in olmocr/work_queue.py orchestrates distributed batch operations across multiple workers. This component handles S3-based locking mechanisms and local lock files to prevent duplicate processing in distributed environments. The MetricsKeeper and WorkerTracker utilities in olmocr/metrics.py maintain live token counts and success-rate statistics throughout the pipeline execution.

Running OlmOCR Locally

To process a single PDF through the pipeline, initialize the requisite arguments and invoke process_pdf asynchronously:

import asyncio
from argparse import Namespace
from olmocr.pipeline import process_pdf

args = Namespace(
    server="http://localhost:8000",
    api_key=None,
    model="olmocr",
    guided_decoding=False,
    target_longest_image_dim=1024,
    max_page_retries=5,
    max_page_error_rate=0.2,
    apply_filter=False,
    markdown=False,
    workspace="/tmp/olmocr_workspace",
)

async def main():
    doc = await process_pdf(args, worker_id=0, pdf_orig_path="s3://my-bucket/sample.pdf")
    print(doc)  # Dolma JSON-L document

asyncio.run(main())

For batch processing workloads, leverage the WorkQueue with multiple concurrent workers:

import asyncio
from argparse import Namespace
from olmocr.work_queue import WorkQueue, LocalBackend
from olmocr.pipeline import worker

args = Namespace(
    server="http://localhost:8000",
    api_key=None,
    model="olmocr",
    guided_decoding=False,
    target_longest_image_dim=1024,
    max_page_retries=5,
    max_page_error_rate=0.2,
    apply_filter=False,
    markdown=True,
    workspace="/tmp/olmocr_workspace",
    max_workers=4,
)

async def run():
    backend = LocalBackend(workspace_path=args.workspace)
    queue = WorkQueue(backend)
    
    await queue.populate_queue(
        work_paths=["data/file1.pdf", "data/file2.pdf"],
        items_per_group=5
    )
    await queue.initialize_queue()
    
    workers = [
        asyncio.create_task(worker(args, queue, worker_id=i))
        for i in range(args.max_workers)
    ]
    await asyncio.gather(*workers)

asyncio.run(run())

Summary

• Eight-stage pipeline: OlmOCR processes PDFs through ingestion, rendering, LLM inference, parsing, rotation correction, fallback recovery, assembly, and output stages.
• Vision-first approach: Pages convert to PNG images before processing through the build_no_anchoring_v4_yaml_prompt via chat completions endpoints.
• Robust error handling: The system implements bounded async semaphores, temperature-based retry logic, and pdftotext fallbacks to maximize extraction success.
• Structured output: Results conform to the Dolma document specification with rich metadata attributes captured from front-matter parsing.
• Distributed ready: The WorkQueue architecture supports S3-backed distributed processing with locking mechanisms to prevent duplicate work.

Frequently Asked Questions

What happens when the LLM fails to extract text from a page?

The pipeline implements a multi-layered recovery system. First, process_page retries the request with rotation corrections if the model flags orientation issues. After exhausting max_page_retries attempts, make_fallback_result invokes pdftotext to extract raw text without vision model processing. This ensures every page yields usable output even when image quality or layout complexity challenges the LLM.

How does OlmOCR handle incorrectly rotated pages?

During the try_single_page execution, the model returns is_rotation_valid status in the front matter. When this flag is false, the process_page function increments the cumulative_rotation counter and re-renders the page at the corrected angle before resubmitting to the LLM. This retry loop continues until the model validates the orientation or the system reaches the configured retry limit.

Can OlmOCR process image files directly, or only PDFs?

The process_pdf function automatically converts JPEG and PNG inputs to PDF format before processing. This conversion happens during the ingestion stage, allowing the pipeline to treat all visual inputs uniformly. The system handles the format transformation transparently, requiring no manual intervention from users working with scanned image documents.

What is the Dolma document format produced by OlmOCR?

The build_dolma_document function generates a JSON-L structured document where the text field contains concatenated page content and the attributes dictionary stores per-page metadata. This metadata includes detected language, rotation angles, table presence flags, and diagram indicators extracted by the FrontMatterParser. The format follows the Dolma corpus specification designed for large-scale NLP training datasets.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}