# How Data-Parallel Inference Works in olmOCR for Processing Multiple PDFs

> Discover how olmOCR uses data-parallel inference to process multiple PDFs. Learn how local vLLM servers distribute page requests across GPUs for faster results.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-06

---

**Data-parallel inference in olmOCR launches a local vLLM server with multiple model replicas across GPUs, enabling the pipeline to distribute individual PDF page requests concurrently across available hardware resources.**

The `allenai/olmocr` repository implements a scalable OCR architecture that leverages **data-parallel inference** to maximize throughput when processing large volumes of PDF documents. By configuring the `--data-parallel-size` parameter, users can deploy replicated model instances that process split pages simultaneously, transforming single-GPU bottlenecks into multi-GPU parallel execution.

## Understanding the vLLM Data-Parallel Architecture

At the core of olmOCR's parallel processing capability lies a local **vLLM** inference server. The `--data-parallel-size` flag, passed during server initialization in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), determines how many **model replicas** the vLLM engine maintains in memory (see lines 1249–1257). Each replica occupies a distinct GPU or GPU group, allowing the system to route incoming requests to any available replica rather than queuing them behind a single model instance.

This architecture decouples request submission from execution. When the pipeline sends a page-level inference request, vLLM dispatches it to the first replica with available capacity. Consequently, multiple PDF pages—potentially from different source documents—undergo **simultaneous OCR inference** without blocking each other, provided sufficient GPU resources exist to support the configured parallel size.

## Configuring GPU Allocation via CLI

You control data-parallel scaling through the `-dp` or `--data-parallel-size` command-line argument when launching the pipeline. The default value is `1`, which maintains a single model replica suitable for single-GPU deployments. To utilize multiple GPUs for parallel inference, increase this value:

```bash
python -m olmocr.pipeline \
    --workspace /path/to/pdfs \
    --data-parallel-size 4 \
    --tensor-parallel-size 1

```

In [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), the `vllm_server_task` function constructs the server launch command by injecting `args.data_parallel_size` directly into the vLLM serve invocation (lines 822–828). The server subprocess starts with the exact replica count specified, allocating one GPU per replica unless combined with tensor parallelism.

## The Page-Level Processing Workflow

olmOCR achieves parallelism by decomposing PDFs into discrete units of work. The pipeline **splits each PDF into individual pages**, then treats every page as an independent inference request. This granular approach allows the system to saturate all available model replicas with steady streams of page-level tasks.

Request concurrency is governed by an asynchronous semaphore named `max_concurrent_requests_limit`, initialized from the `--max_concurrent_requests` CLI flag (default 1600) in lines 84–86 of [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). The `try_single_page` function acquires this semaphore before dispatching each request via `await apost(COMPLETION_URL, …)`, ensuring the pipeline does not overwhelm the vLLM server while still maintaining high throughput across data-parallel replicas.

## Step-by-Step Execution Flow

### Step 1: CLI Argument Parsing

The pipeline parses `--data-parallel-size` into `args.data_parallel_size` during initialization (lines 1249–1257). This value propagates through the configuration object and determines GPU allocation strategy alongside complementary flags like `--tensor-parallel-size`.

### Step 2: vLLM Server Initialization

The `vllm_server_task` function assembles the server command string:

```python
command = f"vllm serve … --data-parallel-size {args.data_parallel_size}"

```

This command launches as a subprocess (lines 822–828), instantiating the specified number of model replicas across available CUDA devices. Each replica loads the full model weights independently, creating isolated inference workers.

### Step 3: Async Request Generation

For every page in every PDF, the pipeline calls `try_single_page`, which constructs a JSON payload containing the base64-encoded page image and structured prompt. The function submits this payload to the local vLLM endpoint, guarded by the `max_concurrent_requests_limit` semaphore to prevent resource exhaustion.

### Step 4: Distributed Inference Execution

Inside the vLLM server, the data-parallel router directs each incoming request to an idle model replica. With `--data-parallel-size` set to N, the system processes up to N pages simultaneously on N distinct GPUs. Results return asynchronously to the pipeline, which aggregates per-page outputs into complete document transcripts.

## Practical Implementation Examples

Deploy data-parallel inference for high-throughput PDF processing using these configurations:

```bash

# Process PDFs using 4-GPU data parallelism

python -m olmocr.pipeline \
    --workspace s3://my-bucket/pdfs \
    --output s3://my-bucket/ocr-results \
    --tensor-parallel-size 1 \
    --data-parallel-size 4 \
    --max_concurrent_requests 2000 \
    --workers 30

```

For programmatic execution, configure the pipeline via `sys.argv` before calling the main entry point:

```python
from olmocr.pipeline import main

if __name__ == "__main__":
    import sys
    sys.argv = [
        "pipeline.py",
        "--workspace", "s3://my-bucket/pdfs",
        "--output", "s3://my-bucket/ocr-results",
        "--data-parallel-size", "2",
        "--max_concurrent_requests", "1000",
    ]
    main()

```

Both examples instantiate vLLM with multiple model replicas, enabling concurrent page processing that scales linearly with GPU count.

## Key Source Files and Functions

The data-parallel implementation spans several critical components:

- **[`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)**: Core orchestration file containing argument parsing (lines 1249–1257), the `vllm_server_task` function that builds the vLLM launch command (lines 822–828), and the `max_concurrent_requests_limit` semaphore definition (lines 84–86) that throttles request concurrency.

- **`try_single_page`**: Async function within [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) responsible for constructing page-level inference requests and submitting them to the vLLM server via HTTP POST operations.

- **[`olmocr/bench/runners/run_olmocr_pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/runners/run_olmocr_pipeline.py)**: Reference implementation demonstrating how benchmark utilities forward vLLM arguments—including data-parallel configuration—to the main pipeline.

## Summary

- **Data-parallel inference** utilizes vLLM's `--data-parallel-size` flag to maintain multiple independent model replicas across GPUs.
- The olmOCR pipeline splits PDFs into pages and submits each as a separate async request, enabling fine-grained parallelism.
- Concurrency is controlled by the `max_concurrent_requests_limit` semaphore, defaulting to 1600 simultaneous requests.
- Configuration occurs through the `--data-parallel-size` CLI argument, parsed in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) and passed to the vLLM server subprocess.
- This architecture scales OCR throughput linearly with available GPU hardware, processing many PDFs simultaneously without accuracy degradation.

## Frequently Asked Questions

### What is the difference between data-parallel and tensor-parallel inference in olmOCR?

**Data-parallel inference** replicates the entire model across multiple GPUs, with each replica handling different requests simultaneously. **Tensor-parallel inference** splits individual layers of a single model across multiple GPUs to accommodate models too large for one device's memory. olmOCR supports both: use `--data-parallel-size` to scale throughput across documents and `--tensor-parallel-size` to scale model capacity across devices.

### How many GPUs do I need for data-parallel processing?

You need at least as many GPUs as your `--data-parallel-size` value. If you set `--data-parallel-size 4`, the system requires four available GPUs, assigning one replica per device. Attempting to allocate more replicas than available GPUs will result in runtime errors during vLLM server initialization.

### Does increasing data-parallel size affect OCR accuracy?

No. Data-parallel inference distributes identical model replicas across GPUs; each replica executes the same weights and inference logic. The parallelism affects only **throughput** and **latency**, not the textual accuracy or formatting of the OCR output. Results remain deterministic regardless of which replica processes a specific page.

### How do I optimize max_concurrent_requests for my hardware?

Set `--max_concurrent_requests` high enough to keep all data-parallel replicas saturated (typically 400–500 requests per replica), but low enough to prevent GPU memory exhaustion. The default value of 1600 suits most 4-GPU data-parallel configurations. Monitor GPU utilization; if utilization drops between batches, increase the concurrency limit. If you encounter CUDA out-of-memory errors, reduce the value or increase `--data-parallel-size` to distribute the load across more replicas.