# How to Perform Multi-Node Cluster Processing with olmOCR: Distributed PDF OCR Guide

> Learn how to perform multi-node cluster processing with olmOCR. This guide details distributed PDF OCR using a scalable S3 work queue for efficient coordination.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**olmOCR scales from a single machine to a compute cluster using a distributed S3 work queue that coordinates workers via lock files and completion flags.**

The allenai/olmocr repository provides a complete distributed processing framework designed for high-volume PDF OCR workloads. By leveraging a shared S3 workspace and the `WorkQueue` abstraction in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py), you can horizontally scale document processing across dozens of GPU nodes without manual work distribution. This guide explains how to configure multi-node cluster processing using either Beaker orchestration or manual multi-host deployment.

## Understanding the Distributed Work Queue Architecture

At the core of olmOCR's scaling mechanism is the `WorkQueue` class, which abstracts PDF grouping, worker coordination, and completion tracking across nodes.

### The WorkQueue Interface

The generic queue interface lives in [[`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py)](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py). This module defines how work items are partitioned, claimed, and marked as complete across distributed workers.

Work items are created by `WorkQueue.populate_queue()`, which groups PDF paths according to the `items_per_group` parameter (default 1) and generates deterministic hashes that serve as unique identifiers for each group.

### Storage Backends: S3Backend vs LocalBackend

Two concrete backends implement the queue interface:

- **LocalBackend**: Uses the local filesystem for single-machine processing
- **S3Backend**: Stores queue state, worker locks, and completion flags in an S3-compatible bucket

For multi-node cluster processing, you must use the **S3Backend**, as it allows all nodes to share the same work index and synchronization state.

## Setting Up the Shared S3 Workspace

Before launching workers, initialize the shared workspace and populate the work queue. According to the olmOCR source code, you must first generate the work index without starting workers:

```bash

# Define your S3 workspace and PDF source

WORKSPACE=s3://my-bucket/olmocr-workspace
PDF_GLOB="s3://my-bucket/pdfs/**/*.pdf"

# Generate the work index (no workers yet)

python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 0 \
  --pages_per_group 1

```

This command calls `WorkQueue.populate_queue()` (see [[`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) lines 740-753), which writes the `work_index_list.csv.zstd` file to your S3 workspace. This index serves as the source of truth for all subsequent worker nodes.

## Launching Multi-Node Workers

olmOCR supports two methods for distributed execution: automated Beaker deployment or manual multi-host configuration.

### Option 1: Automated Beaker Deployment

For AI2 Beaker users, the `submit_beaker_job` function (around line 1010 in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)) automates cluster provisioning:

```bash
python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 20 \
  --pages_per_group 1 \
  --beaker \
  --beaker_gpus 4 \
  --beaker_cluster a100-40g \
  --beaker_priority high

```

**Key parameters:**
- `--beaker`: Activates the Beaker submission mode
- `--beaker_gpus 4`: Allocates 4 GPUs per replica (node)
- `--beaker_cluster a100-40g`: Specifies the Beaker cluster type

The pipeline builds a Beaker experiment specification that spawns multiple replicas. Each replica runs the same command but enters the work-queue loop (lines 698-801 in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)) to pull unique items from the shared S3 queue.

### Option 2: Manual Multi-Host Execution

To run workers on existing infrastructure without Beaker, launch the same command on each node:

```bash

# Configure AWS credentials on each host

export AWS_ACCESS_KEY_ID=your_key_id
export AWS_SECRET_ACCESS_KEY=your_secret_key

# Launch workers on each node

python -m olmocr.pipeline \
  $WORKSPACE \
  --pdfs $PDF_GLOB \
  --workers 10 \
  --pages_per_group 1

```

Each host independently calls `WorkQueue.get_work()` (lines 998-1015 in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)), checks for active locks via `is_worker_lock_taken()`, and claims available items. Because all nodes read from and write to the same S3 prefix, they automatically coordinate without additional configuration.

## How the Queue Prevents Duplicate Processing

The `WorkQueue` implementation guarantees exactly-once processing through a distributed locking protocol:

1. **Worker Lock Creation**: When `get_work()` identifies an available item, it calls `create_worker_lock()` to claim exclusive processing rights
2. **Lock Timeout**: Locks automatically expire after 30 minutes (configurable), ensuring stalled nodes don't permanently block work items
3. **Completion Flags**: After processing, `mark_done()` creates a done flag file (`done_<hash>.flag`) in the `done_flags` directory and deletes the worker lock via `delete_worker_lock()`

This mechanism prevents race conditions where multiple nodes might attempt to process the same PDF group simultaneously.

## Monitoring Progress and Aggregating Results

Track cluster-wide completion using the built-in statistics command:

```bash
python -m olmocr.pipeline \
  $WORKSPACE \
  --print_stats

```

The `print_stats` helper (lines 1055-1120 in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)) scans the `done_flags` directory and result JSONL files to report:
- Total completion percentage
- Token usage statistics
- Skipped or failed PDFs
- Language distribution metrics

Each node writes output JSONL files to `workspace/results/`, allowing you to aggregate results by downloading the entire prefix after the queue empties.

## Summary

- **olmOCR** uses an S3-backed `WorkQueue` to coordinate multi-node cluster processing without a central scheduler
- **Initialize** the workspace with `--workers 0` to generate the work index before launching distributed workers
- **Deploy** via Beaker using `--beaker` flags for automated scaling, or manually launch the pipeline on multiple hosts pointing to the same S3 workspace
- **Prevent conflicts** through automatic worker locks (30-minute timeout) and atomic done flags
- **Monitor** progress using `--print_stats` to view completion rates and token usage across the entire cluster

## Frequently Asked Questions

### How does olmOCR handle node failures during multi-node processing?

If a node fails while processing a work item, its worker lock remains in place but expires after the default 30-minute timeout. Once expired, other healthy nodes will automatically reclaim the work item when they call `get_work()`. The deterministic hashing ensures that even if multiple nodes attempt to process the same PDF group, only the first to complete writes the done flag, and subsequent attempts are skipped.

### Can I use a storage backend other than S3 for distributed processing?

The `WorkQueue` abstraction in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) supports pluggable backends, but the repository currently only implements `S3Backend` and `LocalBackend` for production use. For true multi-node cluster processing, S3 or any S3-compatible object store (MinIO, Ceph) is required, as the coordination depends on atomic object operations and shared visibility across nodes.

### What is the optimal value for `--pages_per_group` in a cluster environment?

The `--pages_per_group` parameter controls granularity. A value of **1** creates maximum parallelism by treating each PDF as a separate work item, ideal for clusters with many nodes. Higher values group multiple PDFs into single work items, reducing S3 API overhead but potentially leaving nodes idle if PDF sizes vary significantly. For homogeneous workloads on large clusters, use 1; for small clusters processing thousands of tiny PDFs, group 10-50 per item.

### How do I verify that all nodes are reading from the same work queue?

All nodes must specify the exact same S3 workspace URL (e.g., `s3://my-bucket/olmocr-workspace/`). Verify configuration by checking that each node's logs reference the same `work_index_list.csv.zstd` hash. The `WorkQueue` loads this index at startup, so mismatched workspace paths will result in nodes processing different datasets or claiming overlapping work items.