How to Configure vLLM Arguments in olmOCR: A Complete Guide

You can configure vLLM parameters in olmOCR by passing standard vllm serve flags directly to the olmocr.pipeline command, which automatically forwards them to either a locally started server or an external endpoint.

The allenai/olmocr repository provides a document OCR pipeline that runs inference through a vLLM backend. Understanding how to configure vLLM arguments in olmOCR allows you to optimize GPU memory usage, adjust token limits, and set up distributed inference across multiple GPUs.

Understanding vLLM Configuration in olmOCR

Where Arguments Are Defined

In olmocr/pipeline.py, the argument parser creates a dedicated vllm_group (lines 49-58) that exposes common vLLM configuration options. This group maps directly to the parameters accepted by the vllm serve command, ensuring compatibility with the underlying inference engine.

Supported vLLM Flags

The pipeline explicitly supports the most frequently used vLLM knobs:

• --gpu-memory-utilization (float): Fraction of VRAM vLLM may pre-allocate for the KV cache.
• --max_model_len (int): Upper token limit for the KV cache (defaults to 16384).
• --tensor-parallel-size or -tp (int): Number of tensor-parallel workers for model sharding.
• --data-parallel-size or -dp (int): Number of data-parallel workers for request batching.
• --port (int): Port on which the local vLLM server will listen (defaults to 30024).

Passing vLLM Arguments to the Pipeline

Standard Configuration Flags

When you invoke python -m olmocr.pipeline, add any supported flag directly to the command line. The parser validates these arguments and passes them to the vLLM subprocess启动器.

For example, to limit GPU memory usage to 85% and enable tensor parallelism across two GPUs:

python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --workers 12 \
    --gpu-memory-utilization 0.85 \
    --tensor-parallel-size 2 \
    --max_model_len 16384

Forwarding Unknown Arguments

The pipeline uses parse_known_args() (lines 72-73) to capture unrecognized arguments in an unknown_args list. This design ensures forward-compatibility with new vLLM releases—you can pass any valid vLLM flag even if olmOCR hasn't explicitly added it to the argument group.

For instance, to set a custom quantization parameter or logging level not listed in the built-in help:

python -m olmocr.pipeline \
    workspace \
    --pdfs list_of_pdfs.txt \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --quantization awq

Local vs. Remote vLLM Servers

Starting a Local vLLM Server

When you omit the --server flag, olmocr/pipeline.py spawns a local vLLM process. The vllm_server_host function (lines 466-475) orchestrates the startup, while vllm_server_task (lines 807-831) constructs the subprocess command by concatenating the user-provided flags into a cmd list. This subprocess runs vllm serve with your specified configuration, integrating automatically with the pipeline's async workflow.

Connecting to External Endpoints

To bypass the local server startup and connect to an already-running vLLM instance (such as DeepInfra or a self-hosted endpoint), use the --server flag:

python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --server http://my.remote.host:8000/v1 \
    --api_key $DEEPINFRA_API_KEY \
    --max_concurrent_requests 400

When using an external server, the pipeline skips the vllm_server_task invocation and routes all inference requests directly to the provided URL.

Practical Configuration Examples

Basic local deployment with optimized memory:

python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --workers 12 \
    --max_concurrent_requests 800 \
    --gpu-memory-utilization 0.85 \
    --max_model_len 16384 \
    --tensor-parallel-size 2 \
    --data-parallel-size 1

Remote endpoint configuration:

python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --server http://localhost:8000/v1 \
    --api_key $API_KEY \
    --max_concurrent_requests 400

Extended context window with additional vLLM flags:

python -m olmocr.pipeline \
    workspace \
    --pdfs list_of_pdfs.txt \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --enforce-eager

Summary

• Argument definition: vLLM flags are defined in olmocr/pipeline.py (lines 49-58) within a dedicated argument group.
• Flexible forwarding: Unknown arguments pass through via parse_known_args() (lines 72-73), supporting any valid vllm serve option.
• Local execution: The pipeline constructs vLLM subprocess commands in vllm_server_task (lines 807-831) when no external server is specified.
• Remote execution: Use --server to connect to existing vLLM endpoints, bypassing local process management.
• Key parameters: Control memory with --gpu-memory-utilization, scaling with --tensor-parallel-size, and context length with --max_model_len.

Frequently Asked Questions

What vLLM arguments are officially supported by olmOCR?

The vllm_group in olmocr/pipeline.py explicitly exposes --gpu-memory-utilization, --max_model_len, --tensor-parallel-size (-tp), --data-parallel-size (-dp), and --port. However, because the pipeline forwards unknown arguments to the vLLM subprocess, you can use any flag supported by your installed vLLM version.

Can I use hyphens instead of underscores in vLLM argument names?

Yes. While the built-in help shows underscores (e.g., --max_model_len), vLLM accepts both hyphenated and underscored forms. The unknown_args mechanism in lines 72-73 passes your exact flag text to the server command, so --max-model-len works identically to --max_model_len.

How do I connect olmOCR to a remote vLLM instance?

Specify the endpoint URL using the --server flag followed by the OpenAI-compatible API URL (typically ending in /v1). You must also provide an --api_key if the remote endpoint requires authentication. When this flag is present, the pipeline skips the local server initialization logic in vllm_server_host.

Where is the vLLM server command constructed in the source code?

The subprocess command assembly occurs in vllm_server_task within olmocr/pipeline.py (lines 807-831). This function builds the cmd list that invokes vllm serve, incorporating all user-provided arguments before executing the process.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}