How Data-Parallel Inference Works in olmOCR for Processing Multiple PDFs

Data-parallel inference in olmOCR launches a local vLLM server with multiple model replicas across GPUs, enabling the pipeline to distribute individual PDF page requests concurrently across available hardware resources.

The allenai/olmocr repository implements a scalable OCR architecture that leverages data-parallel inference to maximize throughput when processing large volumes of PDF documents. By configuring the --data-parallel-size parameter, users can deploy replicated model instances that process split pages simultaneously, transforming single-GPU bottlenecks into multi-GPU parallel execution.

Understanding the vLLM Data-Parallel Architecture

At the core of olmOCR's parallel processing capability lies a local vLLM inference server. The --data-parallel-size flag, passed during server initialization in olmocr/pipeline.py, determines how many model replicas the vLLM engine maintains in memory (see lines 1249–1257). Each replica occupies a distinct GPU or GPU group, allowing the system to route incoming requests to any available replica rather than queuing them behind a single model instance.

This architecture decouples request submission from execution. When the pipeline sends a page-level inference request, vLLM dispatches it to the first replica with available capacity. Consequently, multiple PDF pages—potentially from different source documents—undergo simultaneous OCR inference without blocking each other, provided sufficient GPU resources exist to support the configured parallel size.

Configuring GPU Allocation via CLI

You control data-parallel scaling through the -dp or --data-parallel-size command-line argument when launching the pipeline. The default value is 1, which maintains a single model replica suitable for single-GPU deployments. To utilize multiple GPUs for parallel inference, increase this value:

python -m olmocr.pipeline \
    --workspace /path/to/pdfs \
    --data-parallel-size 4 \
    --tensor-parallel-size 1

In olmocr/pipeline.py, the vllm_server_task function constructs the server launch command by injecting args.data_parallel_size directly into the vLLM serve invocation (lines 822–828). The server subprocess starts with the exact replica count specified, allocating one GPU per replica unless combined with tensor parallelism.

The Page-Level Processing Workflow

olmOCR achieves parallelism by decomposing PDFs into discrete units of work. The pipeline splits each PDF into individual pages, then treats every page as an independent inference request. This granular approach allows the system to saturate all available model replicas with steady streams of page-level tasks.

Request concurrency is governed by an asynchronous semaphore named max_concurrent_requests_limit, initialized from the --max_concurrent_requests CLI flag (default 1600) in lines 84–86 of olmocr/pipeline.py. The try_single_page function acquires this semaphore before dispatching each request via await apost(COMPLETION_URL, …), ensuring the pipeline does not overwhelm the vLLM server while still maintaining high throughput across data-parallel replicas.

Step-by-Step Execution Flow

Step 1: CLI Argument Parsing

The pipeline parses --data-parallel-size into args.data_parallel_size during initialization (lines 1249–1257). This value propagates through the configuration object and determines GPU allocation strategy alongside complementary flags like --tensor-parallel-size.

Step 2: vLLM Server Initialization

The vllm_server_task function assembles the server command string:

command = f"vllm serve … --data-parallel-size {args.data_parallel_size}"

This command launches as a subprocess (lines 822–828), instantiating the specified number of model replicas across available CUDA devices. Each replica loads the full model weights independently, creating isolated inference workers.

Step 3: Async Request Generation

For every page in every PDF, the pipeline calls try_single_page, which constructs a JSON payload containing the base64-encoded page image and structured prompt. The function submits this payload to the local vLLM endpoint, guarded by the max_concurrent_requests_limit semaphore to prevent resource exhaustion.

Step 4: Distributed Inference Execution

Inside the vLLM server, the data-parallel router directs each incoming request to an idle model replica. With --data-parallel-size set to N, the system processes up to N pages simultaneously on N distinct GPUs. Results return asynchronously to the pipeline, which aggregates per-page outputs into complete document transcripts.

Practical Implementation Examples

Deploy data-parallel inference for high-throughput PDF processing using these configurations:

# Process PDFs using 4-GPU data parallelism

python -m olmocr.pipeline \
    --workspace s3://my-bucket/pdfs \
    --output s3://my-bucket/ocr-results \
    --tensor-parallel-size 1 \
    --data-parallel-size 4 \
    --max_concurrent_requests 2000 \
    --workers 30

For programmatic execution, configure the pipeline via sys.argv before calling the main entry point:

from olmocr.pipeline import main

if __name__ == "__main__":
    import sys
    sys.argv = [
        "pipeline.py",
        "--workspace", "s3://my-bucket/pdfs",
        "--output", "s3://my-bucket/ocr-results",
        "--data-parallel-size", "2",
        "--max_concurrent_requests", "1000",
    ]
    main()

Both examples instantiate vLLM with multiple model replicas, enabling concurrent page processing that scales linearly with GPU count.

Key Source Files and Functions

The data-parallel implementation spans several critical components:

• olmocr/pipeline.py: Core orchestration file containing argument parsing (lines 1249–1257), the vllm_server_task function that builds the vLLM launch command (lines 822–828), and the max_concurrent_requests_limit semaphore definition (lines 84–86) that throttles request concurrency.

• try_single_page: Async function within pipeline.py responsible for constructing page-level inference requests and submitting them to the vLLM server via HTTP POST operations.

• olmocr/bench/runners/run_olmocr_pipeline.py: Reference implementation demonstrating how benchmark utilities forward vLLM arguments—including data-parallel configuration—to the main pipeline.

Summary

• Data-parallel inference utilizes vLLM's --data-parallel-size flag to maintain multiple independent model replicas across GPUs.
• The olmOCR pipeline splits PDFs into pages and submits each as a separate async request, enabling fine-grained parallelism.
• Concurrency is controlled by the max_concurrent_requests_limit semaphore, defaulting to 1600 simultaneous requests.
• Configuration occurs through the --data-parallel-size CLI argument, parsed in olmocr/pipeline.py and passed to the vLLM server subprocess.
• This architecture scales OCR throughput linearly with available GPU hardware, processing many PDFs simultaneously without accuracy degradation.

Frequently Asked Questions

What is the difference between data-parallel and tensor-parallel inference in olmOCR?

Data-parallel inference replicates the entire model across multiple GPUs, with each replica handling different requests simultaneously. Tensor-parallel inference splits individual layers of a single model across multiple GPUs to accommodate models too large for one device's memory. olmOCR supports both: use --data-parallel-size to scale throughput across documents and --tensor-parallel-size to scale model capacity across devices.

How many GPUs do I need for data-parallel processing?

You need at least as many GPUs as your --data-parallel-size value. If you set --data-parallel-size 4, the system requires four available GPUs, assigning one replica per device. Attempting to allocate more replicas than available GPUs will result in runtime errors during vLLM server initialization.

Does increasing data-parallel size affect OCR accuracy?

No. Data-parallel inference distributes identical model replicas across GPUs; each replica executes the same weights and inference logic. The parallelism affects only throughput and latency, not the textual accuracy or formatting of the OCR output. Results remain deterministic regardless of which replica processes a specific page.

How do I optimize max_concurrent_requests for my hardware?

Set --max_concurrent_requests high enough to keep all data-parallel replicas saturated (typically 400–500 requests per replica), but low enough to prevent GPU memory exhaustion. The default value of 1600 suits most 4-GPU data-parallel configurations. Monitor GPU utilization; if utilization drops between batches, increase the concurrency limit. If you encounter CUDA out-of-memory errors, reduce the value or increase --data-parallel-size to distribute the load across more replicas.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}