How to Perform Multi-Node Cluster Processing with olmOCR: Distributed PDF OCR Guide

olmOCR scales from a single machine to a compute cluster using a distributed S3 work queue that coordinates workers via lock files and completion flags.

The allenai/olmocr repository provides a complete distributed processing framework designed for high-volume PDF OCR workloads. By leveraging a shared S3 workspace and the WorkQueue abstraction in olmocr/work_queue.py, you can horizontally scale document processing across dozens of GPU nodes without manual work distribution. This guide explains how to configure multi-node cluster processing using either Beaker orchestration or manual multi-host deployment.

Understanding the Distributed Work Queue Architecture

At the core of olmOCR's scaling mechanism is the WorkQueue class, which abstracts PDF grouping, worker coordination, and completion tracking across nodes.

The WorkQueue Interface

The generic queue interface lives in [olmocr/work_queue.py](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py). This module defines how work items are partitioned, claimed, and marked as complete across distributed workers.

Work items are created by WorkQueue.populate_queue(), which groups PDF paths according to the items_per_group parameter (default 1) and generates deterministic hashes that serve as unique identifiers for each group.

Storage Backends: S3Backend vs LocalBackend

Two concrete backends implement the queue interface:

• LocalBackend: Uses the local filesystem for single-machine processing
• S3Backend: Stores queue state, worker locks, and completion flags in an S3-compatible bucket

For multi-node cluster processing, you must use the S3Backend, as it allows all nodes to share the same work index and synchronization state.

Setting Up the Shared S3 Workspace

Before launching workers, initialize the shared workspace and populate the work queue. According to the olmOCR source code, you must first generate the work index without starting workers:

# Define your S3 workspace and PDF source

WORKSPACE=s3://my-bucket/olmocr-workspace
PDF_GLOB="s3://my-bucket/pdfs/**/*.pdf"

# Generate the work index (no workers yet)

python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 0 \
  --pages_per_group 1

This command calls WorkQueue.populate_queue() (see [olmocr/pipeline.py](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) lines 740-753), which writes the work_index_list.csv.zstd file to your S3 workspace. This index serves as the source of truth for all subsequent worker nodes.

Launching Multi-Node Workers

olmOCR supports two methods for distributed execution: automated Beaker deployment or manual multi-host configuration.

Option 1: Automated Beaker Deployment

For AI2 Beaker users, the submit_beaker_job function (around line 1010 in olmocr/pipeline.py) automates cluster provisioning:

python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 20 \
  --pages_per_group 1 \
  --beaker \
  --beaker_gpus 4 \
  --beaker_cluster a100-40g \
  --beaker_priority high

Key parameters:

• --beaker: Activates the Beaker submission mode
• --beaker_gpus 4: Allocates 4 GPUs per replica (node)
• --beaker_cluster a100-40g: Specifies the Beaker cluster type

The pipeline builds a Beaker experiment specification that spawns multiple replicas. Each replica runs the same command but enters the work-queue loop (lines 698-801 in pipeline.py) to pull unique items from the shared S3 queue.

Option 2: Manual Multi-Host Execution

To run workers on existing infrastructure without Beaker, launch the same command on each node:

# Configure AWS credentials on each host

export AWS_ACCESS_KEY_ID=your_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_key

# Launch workers on each node

python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 10 \
  --pages_per_group 1

Each host independently calls WorkQueue.get_work() (lines 998-1015 in pipeline.py), checks for active locks via is_worker_lock_taken(), and claims available items. Because all nodes read from and write to the same S3 prefix, they automatically coordinate without additional configuration.

How the Queue Prevents Duplicate Processing

The WorkQueue implementation guarantees exactly-once processing through a distributed locking protocol:

1. Worker Lock Creation: When get_work() identifies an available item, it calls create_worker_lock() to claim exclusive processing rights
2. Lock Timeout: Locks automatically expire after 30 minutes (configurable), ensuring stalled nodes don't permanently block work items
3. Completion Flags: After processing, mark_done() creates a done flag file (done_<hash>.flag) in the done_flags directory and deletes the worker lock via delete_worker_lock()

This mechanism prevents race conditions where multiple nodes might attempt to process the same PDF group simultaneously.

Monitoring Progress and Aggregating Results

Track cluster-wide completion using the built-in statistics command:

python -m olmocr.pipeline \
  $WORKSPACE \
  --print_stats

The print_stats helper (lines 1055-1120 in pipeline.py) scans the done_flags directory and result JSONL files to report:

• Total completion percentage
• Token usage statistics
• Skipped or failed PDFs
• Language distribution metrics

Each node writes output JSONL files to workspace/results/, allowing you to aggregate results by downloading the entire prefix after the queue empties.

Summary

• olmOCR uses an S3-backed WorkQueue to coordinate multi-node cluster processing without a central scheduler
• Initialize the workspace with --workers 0 to generate the work index before launching distributed workers
• Deploy via Beaker using --beaker flags for automated scaling, or manually launch the pipeline on multiple hosts pointing to the same S3 workspace
• Prevent conflicts through automatic worker locks (30-minute timeout) and atomic done flags
• Monitor progress using --print_stats to view completion rates and token usage across the entire cluster

Frequently Asked Questions

How does olmOCR handle node failures during multi-node processing?

If a node fails while processing a work item, its worker lock remains in place but expires after the default 30-minute timeout. Once expired, other healthy nodes will automatically reclaim the work item when they call get_work(). The deterministic hashing ensures that even if multiple nodes attempt to process the same PDF group, only the first to complete writes the done flag, and subsequent attempts are skipped.

Can I use a storage backend other than S3 for distributed processing?

The WorkQueue abstraction in olmocr/work_queue.py supports pluggable backends, but the repository currently only implements S3Backend and LocalBackend for production use. For true multi-node cluster processing, S3 or any S3-compatible object store (MinIO, Ceph) is required, as the coordination depends on atomic object operations and shared visibility across nodes.

What is the optimal value for --pages_per_group in a cluster environment?

The --pages_per_group parameter controls granularity. A value of 1 creates maximum parallelism by treating each PDF as a separate work item, ideal for clusters with many nodes. Higher values group multiple PDFs into single work items, reducing S3 API overhead but potentially leaving nodes idle if PDF sizes vary significantly. For homogeneous workloads on large clusters, use 1; for small clusters processing thousands of tiny PDFs, group 10-50 per item.

How do I verify that all nodes are reading from the same work queue?

All nodes must specify the exact same S3 workspace URL (e.g., s3://my-bucket/olmocr-workspace/). Verify configuration by checking that each node's logs reference the same work_index_list.csv.zstd hash. The WorkQueue loads this index at startup, so mismatched workspace paths will result in nodes processing different datasets or claiming overlapping work items.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}