# How to Configure vLLM Arguments in olmOCR: A Complete Guide

> Configure vLLM arguments in olmOCR to optimize performance. Learn how to pass flags directly to the olmocr.pipeline command for local or external server integration.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**You can configure vLLM parameters in olmOCR by passing standard `vllm serve` flags directly to the `olmocr.pipeline` command, which automatically forwards them to either a locally started server or an external endpoint.**

The `allenai/olmocr` repository provides a document OCR pipeline that runs inference through a vLLM backend. Understanding how to configure vLLM arguments in olmOCR allows you to optimize GPU memory usage, adjust token limits, and set up distributed inference across multiple GPUs.

## Understanding vLLM Configuration in olmOCR

### Where Arguments Are Defined

In [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), the argument parser creates a dedicated `vllm_group` (lines 49-58) that exposes common vLLM configuration options. This group maps directly to the parameters accepted by the `vllm serve` command, ensuring compatibility with the underlying inference engine.

### Supported vLLM Flags

The pipeline explicitly supports the most frequently used vLLM knobs:

- **`--gpu-memory-utilization`** (float): Fraction of VRAM vLLM may pre-allocate for the KV cache.
- **`--max_model_len`** (int): Upper token limit for the KV cache (defaults to 16384).
- **`--tensor-parallel-size`** or `-tp` (int): Number of tensor-parallel workers for model sharding.
- **`--data-parallel-size`** or `-dp` (int): Number of data-parallel workers for request batching.
- **`--port`** (int): Port on which the local vLLM server will listen (defaults to 30024).

## Passing vLLM Arguments to the Pipeline

### Standard Configuration Flags

When you invoke `python -m olmocr.pipeline`, add any supported flag directly to the command line. The parser validates these arguments and passes them to the vLLM subprocess启动器.

For example, to limit GPU memory usage to 85% and enable tensor parallelism across two GPUs:

```bash
python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --workers 12 \
    --gpu-memory-utilization 0.85 \
    --tensor-parallel-size 2 \
    --max_model_len 16384

```

### Forwarding Unknown Arguments

The pipeline uses `parse_known_args()` (lines 72-73) to capture unrecognized arguments in an `unknown_args` list. This design ensures forward-compatibility with new vLLM releases—you can pass any valid vLLM flag even if olmOCR hasn't explicitly added it to the argument group.

For instance, to set a custom quantization parameter or logging level not listed in the built-in help:

```bash
python -m olmocr.pipeline \
    workspace \
    --pdfs list_of_pdfs.txt \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --quantization awq

```

## Local vs. Remote vLLM Servers

### Starting a Local vLLM Server

When you omit the `--server` flag, [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) spawns a local vLLM process. The `vllm_server_host` function (lines 466-475) orchestrates the startup, while `vllm_server_task` (lines 807-831) constructs the subprocess command by concatenating the user-provided flags into a `cmd` list. This subprocess runs `vllm serve` with your specified configuration, integrating automatically with the pipeline's async workflow.

### Connecting to External Endpoints

To bypass the local server startup and connect to an already-running vLLM instance (such as DeepInfra or a self-hosted endpoint), use the `--server` flag:

```bash
python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --server http://my.remote.host:8000/v1 \
    --api_key $DEEPINFRA_API_KEY \
    --max_concurrent_requests 400

```

When using an external server, the pipeline skips the `vllm_server_task` invocation and routes all inference requests directly to the provided URL.

## Practical Configuration Examples

**Basic local deployment with optimized memory:**

```bash
python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --workers 12 \
    --max_concurrent_requests 800 \
    --gpu-memory-utilization 0.85 \
    --max_model_len 16384 \
    --tensor-parallel-size 2 \
    --data-parallel-size 1

```

**Remote endpoint configuration:**

```bash
python -m olmocr.pipeline \
    s3://my-bucket/workspace \
    --pdfs s3://my-bucket/pdfs/*.pdf \
    --server http://localhost:8000/v1 \
    --api_key $API_KEY \
    --max_concurrent_requests 400

```

**Extended context window with additional vLLM flags:**

```bash
python -m olmocr.pipeline \
    workspace \
    --pdfs list_of_pdfs.txt \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --enforce-eager

```

## Summary

- **Argument definition**: vLLM flags are defined in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 49-58) within a dedicated argument group.
- **Flexible forwarding**: Unknown arguments pass through via `parse_known_args()` (lines 72-73), supporting any valid `vllm serve` option.
- **Local execution**: The pipeline constructs vLLM subprocess commands in `vllm_server_task` (lines 807-831) when no external server is specified.
- **Remote execution**: Use `--server` to connect to existing vLLM endpoints, bypassing local process management.
- **Key parameters**: Control memory with `--gpu-memory-utilization`, scaling with `--tensor-parallel-size`, and context length with `--max_model_len`.

## Frequently Asked Questions

### What vLLM arguments are officially supported by olmOCR?

The `vllm_group` in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) explicitly exposes `--gpu-memory-utilization`, `--max_model_len`, `--tensor-parallel-size` (`-tp`), `--data-parallel-size` (`-dp`), and `--port`. However, because the pipeline forwards unknown arguments to the vLLM subprocess, you can use any flag supported by your installed vLLM version.

### Can I use hyphens instead of underscores in vLLM argument names?

Yes. While the built-in help shows underscores (e.g., `--max_model_len`), vLLM accepts both hyphenated and underscored forms. The `unknown_args` mechanism in lines 72-73 passes your exact flag text to the server command, so `--max-model-len` works identically to `--max_model_len`.

### How do I connect olmOCR to a remote vLLM instance?

Specify the endpoint URL using the `--server` flag followed by the OpenAI-compatible API URL (typically ending in `/v1`). You must also provide an `--api_key` if the remote endpoint requires authentication. When this flag is present, the pipeline skips the local server initialization logic in `vllm_server_host`.

### Where is the vLLM server command constructed in the source code?

The subprocess command assembly occurs in `vllm_server_task` within [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 807-831). This function builds the `cmd` list that invokes `vllm serve`, incorporating all user-provided arguments before executing the process.