Structure of the allenai/olmocr Codebase: Module Breakdown and Architecture Guide

The allenai/olmocr repository is organized as a modular Python package where olmocr/pipeline.py orchestrates PDF-to-text conversion, olmocr/work_queue.py manages distributed job scheduling, and specialized subdirectories handle rendering, filtering, training, and benchmarking.

The allenai/olmocr codebase is a Python toolkit designed to convert PDFs and image-based documents into clean Markdown using vision-language models. Understanding the structure of the allenai/olmocr code reveals a deliberate layered architecture that separates I/O operations, inference logic, and post-processing into distinct, swappable modules. This organization enables scalable OCR processing across both local workstations and distributed S3-backed environments.

Top-Level Directory Structure

The repository is structured as a single Python package named olmocr with specialized submodules for each stage of the document processing pipeline:

olmocr/
├── bench/                # Benchmark suite & helper scripts

├── data/                 # Dataset-generation and PDF rendering utilities

├── filter/               # Language-filtering and coherency checks

├── synth/                # Synthetic data generation (HTML → PDF)

├── train/                # Training / RL-Trainer code

├── viewer/               # Dolma-viewer for visualising OCR results

├── work_queue.py         # Core work-queue abstractions (local & S3 back-ends)

├── pipeline.py          # Main inference pipeline (orchestrates rendering, prompting, VLLM calls)

├── metrics.py            # Token-usage tracking & worker statistics

├── image_utils.py        # Image-to-PDF conversion helpers

├── s3_utils.py           # S3 download/upload helpers

├── prompts/              # Prompt-construction utilities

├── repeatdetect.py       # Page-repeat detection logic

├── version.py            # Package version

└── __init__.py           # Package entry-point (`olmocr` command)

Core Pipeline Orchestration

The olmocr/pipeline.py file serves as the central nervous system of the codebase. It implements the end-to-end processing flow that coordinates all other modules, starting with a set of imports that wire together the rendering, prompting, and inference components at lines 1-53.

The pipeline executes the following sequence:

1. PDF Rendering – Calls render_pdf_to_base64png from olmocr/data/renderpdf.py to convert pages to base64-encoded PNGs.
2. Prompt Construction – Uses build_no_anchoring_v4_yaml_prompt from olmocr/prompts/prompts.py to format inputs for the VLLM.
3. Inference Execution – Dispatches async HTTP requests via httpx using the apost helper in the try_single_page coroutine.
4. Response Parsing – Processes VLLM outputs into structured PageResponse dataclasses.
5. Output Writing – Generates Markdown files and Dolma-compatible JSON for downstream indexing.

Work Queue and Distributed Processing

The olmocr/work_queue.py module defines the infrastructure for parallel document processing across multiple workers. It implements three key classes:

• WorkQueue – Generic interface for scheduling page-group jobs.
• LocalBackend – File-system-based queue for single-machine processing.
• S3Backend – Cloud-backed queue for distributed processing across compute clusters.

These backends enable the pipeline to scale from local laptop runs to massive cloud deployments without changing the core inference logic.

Metrics and Monitoring

The olmocr/metrics.py module provides observability into the OCR process through two primary classes:

• MetricsKeeper – Tracks token usage across all VLLM requests.
• WorkerTracker – Monitors concurrency statistics and worker health.

These are instantiated globally in pipeline.py at lines 78-80, allowing real-time tracking of processing costs and throughput:

from olmocr.metrics import MetricsKeeper, WorkerTracker

metrics = MetricsKeeper()
worker_tracker = WorkerTracker()

Data Processing and PDF Rendering

The olmocr/data/ subdirectory contains utilities for document preparation and silver-training data generation:

• renderpdf.py – Implements render_pdf_to_base64png, the core function for converting PDF pages to base64-encoded PNGs suitable for vision-language models.
• prepare_workspace.py – Constructs processing workspaces for managing intermediate files.
• buildsilver.py – Builds silver-training datasets from processed documents.

Prompt Engineering and Anchor Text

The olmocr/prompts/ directory handles the construction of VLLM inputs:

• prompts.py – Contains build_no_anchoring_v4_yaml_prompt and other prompt builders that format document images with instructions.
• anchor.py – Implements get_anchor_text for extracting anchor text from PDFs to improve model context.

Quality Filtering and Coherency Checks

The olmocr/filter/ subdirectory ensures output quality through:

• filter.py – Language detection, spam identification, and form-field filtering.
• coherency.py – Text-coherency heuristics that validate OCR output correctness.

Training and Synthetic Data Generation

The training infrastructure resides in olmocr/train/ and olmocr/synth/:

• train/train.py – Fine-tuning entry point for the vision-language model.
• train/grpo_train.py – Implements GRPO (Generalized Reward Policy Optimization) reinforcement learning trainer.
• synth/mine_html_templates.py – Generates synthetic training data by converting HTML templates to PDFs.

Benchmarking and Visualization Utilities

The codebase includes comprehensive evaluation tools:

• olmocr/bench/benchmark.py – Full benchmark harness with table-parsing and KaTeX rendering for comparing OCR system accuracy.
• olmocr/viewer/dolmaviewer.py – Dolma-style HTML viewer for visualizing extracted text alongside source images.

How the Components Interact

The structure of the allenai/olmocr code follows a clear data flow that isolates I/O, inference, and post-processing:

CLI → Pipeline → (Render → Prompt → VLLM) → Parse → Workspace
      ↑                                   ↓
   Work-Queue ↔ S3 Utils                Metrics / Logging

When processing a document, the pipeline:

1. Registers the PDF path with the WorkQueue (either LocalBackend or S3Backend).
2. Renders pages using render_pdf_to_base64png with optional rotation handling.
3. Constructs prompts via build_no_anchoring_v4_yaml_prompt and optional anchor text extraction.
4. Executes VLLM calls with temperature scheduling (TEMPERATURE_BY_ATTEMPT) for retry logic.
5. Tracks token consumption via the global metrics object.
6. Writes Markdown and Dolma JSON outputs to the workspace.

Summary

• The allenai/olmocr codebase is organized as a single Python package with modular subdirectories for distinct concerns.
• olmocr/pipeline.py serves as the central orchestrator, importing and coordinating rendering, prompting, and inference components.
• olmocr/work_queue.py provides abstractions for both local and distributed (S3-backed) job scheduling.
• Specialized modules in data/, filter/, train/, and bench/ handle specific tasks like PDF rendering, quality control, model fine-tuning, and benchmarking.
• The architecture separates I/O operations from inference logic, enabling easy swapping of components such as different VLLM backends or filtering strategies.

Frequently Asked Questions

What is the main entry point for the allenai/olmocr package?

The main entry point is olmocr/__init__.py, which exposes the olmocr console script. When installed, the olmocr command simply calls python -m olmocr.pipeline, passing user arguments to the pipeline orchestrator in olmocr/pipeline.py.

How does the codebase handle distributed processing across multiple machines?

Distributed processing is handled by olmocr/work_queue.py, which defines LocalBackend for single-machine execution and S3Backend for cloud-based processing. The WorkQueue class manages page-group jobs across workers, allowing the same pipeline code to run on a laptop or a large compute cluster without modification.

Where is the PDF rendering logic implemented in the allenai/olmocr structure?

PDF rendering is implemented in olmocr/data/renderpdf.py, specifically in the render_pdf_to_base64png function. This utility converts PDF pages into base64-encoded PNGs suitable for vision-language model consumption, handling rotation and other preprocessing steps before the images are sent to the VLLM.

How does the pipeline manage retries and temperature scheduling for VLLM calls?

The pipeline manages retries through the try_single_page coroutine in olmocr/pipeline.py, which uses a TEMPERATURE_BY_ATTEMPT dictionary to vary the sampling temperature on each retry. Token usage and worker statistics are tracked globally via the MetricsKeeper and WorkerTracker classes instantiated in olmocr/metrics.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}