# How to Use olmOCR with a Remote Inference Server: A Complete Configuration Guide

> Scale olmOCR to millions of PDFs by delegating inference to an OpenAI compatible server. Learn how to configure olmOCR with remote inference using the --server flag.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**You can scale olmOCR to millions of PDFs by delegating vision-language model inference to any server implementing the OpenAI Chat Completions API using the `--server` and `--api_key` flags, while the pipeline automatically handles PDF rendering, prompt construction, and error recovery.**

The [allenai/olmocr](https://github.com/allenai/olmocr) repository provides a production-ready pipeline that separates document processing from model inference. By pointing the pipeline at a remote inference server—such as a vLLM instance or managed API provider—you can distribute GPU-intensive work across specialized hardware while the orchestration logic handles PDF parsing, image encoding, and result assembly.

## How the Remote Inference Architecture Works

When you configure olmOCR with a remote server, the pipeline executes an eight-step workflow that transforms raw PDFs into structured Dolma documents.

### PDF Rendering and Encoding

First, `render_pdf_to_base64png` in [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py) converts each PDF page into a base64-encoded PNG image. This function handles DPI scaling, color space conversion, and memory-efficient streaming to support documents with thousands of pages.

### Prompt Construction

Next, `build_page_query` (located in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py)) assembles the chat completion payload. It injects the base64 image alongside the system prompt generated by `build_no_anchoring_v4_yaml_prompt` from [`olmocr/prompts/prompts.py`](https://github.com/allenai/olmocr/blob/main/olmocr/prompts/prompts.py). The resulting JSON payload follows the OpenAI vision format, with the image URL set to `data:image/png;base64,{encoded_string}`.

### HTTP Transport and Authentication

The `apost` async function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) opens a low-level TCP connection to the remote server. When you provide `--api_key`, the function adds an `Authorization: Bearer <key>` header to every POST request. The target URL must end with `/chat/completions`, and the pipeline expects a JSON response containing `usage` token counts and `choices[0].message.content`.

### Response Handling and Fallbacks

`try_single_page` parses the model response, extracts the document structure into a `PageResponse` object, and records metrics via `MetricsKeeper`. If the model returns a rotation error, the pipeline automatically retries with corrected angles. When servers return persistent errors or context-length exceeded messages, the system falls back to `make_fallback_result`, which uses `pdftotext` for pure text extraction.

### Document Assembly and Storage

Finally, `build_dolma_document` merges all page results into a single JSONL record with per-page metadata (detected language, rotation angles, table structures). The `s3_utils` module writes completed documents to your workspace path, whether local filesystem or S3.

## Essential Configuration Flags

To connect olmOCR to a remote inference server, you need three key parameters:

- **`--server`** – The base URL of the remote endpoint (e.g., `http://localhost:8000` or `https://api.deepinfra.com/v1/openai`). The pipeline appends `/chat/completions` automatically.
- **`--api_key`** – Optional bearer token for authenticated endpoints. The pipeline includes this as `Authorization: Bearer <key>` in HTTP headers.
- **`--max_concurrent_requests`** – Limits parallel HTTP connections to prevent overwhelming rate-limited services. Adjust this based on your remote server's capacity.
- **`--workers`** – Local process pool size for PDF preprocessing tasks (rendering and prompt building), which run independently of the inference requests.

## End-to-End Configuration Examples

### Connecting to a Public vLLM Server

The simplest deployment points olmOCR at an existing vLLM instance hosting the `allenai/olmOCR-2-7B-1025-FP8` model:

```bash
python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --server http://my-vllm-instance.example.com:8000 \
  --max_concurrent_requests 300 \
  --workers 20

```

This configuration streams PDFs from S3, sends each page to the remote server, and writes JSONL results back to `s3://my-bucket/olmocr-workspace/results/`.

### Authenticated Providers (DeepInfra)

For commercial APIs requiring authentication, export your key and reference it via `--api_key`:

```bash
export DEEPINFRA_API_KEY=sk-xxxxxxx

python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model deepinfra/olmocr-2b \
  --server https://api.deepinfra.com/v1/openai \
  --api_key $DEEPINFRA_API_KEY \
  --max_concurrent_requests 100 \
  --workers 10

```

The pipeline automatically injects the bearer token into every request to `https://api.deepinfra.com/v1/openai/chat/completions`.

### Local vLLM Server with Distributed Workers

You can host the vLLM server separately and connect multiple olmOCR workers to it:

1. **Start the inference server (one terminal):**

```bash
python -m olmocr.pipeline \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --port 8000 \
  --launch_vllm

```

2. **Run processing workers (other terminals):**

```bash
python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --server http://127.0.0.1:8000 \
  --workers 40 \
  --max_concurrent_requests 800

```

The `--launch_vllm` flag triggers `vllm_server_task` in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), which initializes the model with optimal tensor parallelism settings.

### Generating Markdown Output

To produce human-readable `.md` files alongside the JSONL records:

```bash
python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --server http://my-vllm-instance.example.com:8000 \
  --markdown \
  --workers 15

```

When `--markdown` is set, the pipeline writes cleaned text to `<workspace>/markdown/` using `get_markdown_path` for filename generation.

## Reusing Pipeline Components in Python

If you need to call the remote server outside the full pipeline context, import the `apost` helper directly:

```python
import asyncio
import json
from olmocr.pipeline import apost

async def remote_olmocr(query: dict, server: str, api_key: str | None = None):
    """
    Send a single page query to the remote inference server.
    
    Args:
        query: Dictionary containing the OpenAI-compatible chat completion payload
        server: Base URL of the inference endpoint
        api_key: Optional bearer token for authentication
    """
    url = f"{server.rstrip('/')}/chat/completions"
    status, body = await apost(url, json_data=query, api_key=api_key)
    
    if status != 200:
        raise RuntimeError(f"Server returned HTTP {status}: {body}")
    
    result = json.loads(body)
    return result["choices"][0]["message"]["content"]

# Usage example

async def main():
    query = {
        "model": "allenai/olmOCR-2-7B-1025-FP8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Convert this page to markdown"},
                    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
                ]
            }
        ]
    }
    content = await remote_olmocr(query, "http://localhost:8000")
    print(content)

asyncio.run(main())

```

This low-level approach uses the same `apost` function that the full pipeline relies on for connection pooling, timeout handling, and retry logic.

## Summary

- Use the **`--server`** flag to point olmOCR at any OpenAI-compatible Chat Completions endpoint, including vLLM and commercial providers.
- Authenticate private endpoints by passing **`--api_key`**, which the pipeline converts to `Authorization: Bearer` headers in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py).
- The pipeline automatically encodes PDF pages via `render_pdf_to_base64png` and constructs vision-language prompts through `build_page_query`.
- Automatic backoff and rotation correction are handled by `try_single_page`, with fallback to `pdftotext` if the remote server fails persistently.
- Scale horizontally by running one vLLM server with `--launch_vllm` and connecting multiple olmOCR worker processes via `--server`.

## Frequently Asked Questions

### What API format must the remote inference server implement?

The remote server must expose an OpenAI-compatible Chat Completions endpoint at `/chat/completions`. It should accept JSON POST requests with `model`, `messages`, and optional `max_tokens` parameters, returning responses containing `choices[0].message.content` and `usage` metadata. vLLM, TGI, and most commercial providers (DeepInfra, OpenAI, Together) support this format natively.

### How does olmOCR handle authentication for private inference endpoints?

When you provide the `--api_key` flag, the `apost` function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) automatically adds an `Authorization: Bearer <token>` header to every HTTP request. This supports standard JWT authentication used by private vLLM deployments and cloud API providers without modifying the underlying code.

### What happens if the remote server returns errors or timeouts?

The pipeline implements exponential backoff in `try_single_page_with_backoff` for transient network failures. For specific errors like rotation mismatches, it automatically retries with corrected angles. If the server returns persistent errors (context length exceeded, repeated 5xx codes), the pipeline triggers `make_fallback_result` to extract text using `pdftotext`, ensuring the workflow completes even when inference fails.

### Can I use olmOCR with standard cloud LLMs like GPT-4 or Claude?

While technically possible if the provider supports image inputs via the Chat Completions API, olmOCR is optimized for the `allenai/olmOCR-2-7B-1025-FP8` model architecture. Standard cloud LLMs may not follow the specific prompt templates defined in [`olmocr/prompts/prompts.py`](https://github.com/allenai/olmocr/blob/main/olmocr/prompts/prompts.py) or return the structured YAML front-matter that `try_single_page` expects for document assembly. For best results, use a vLLM server hosting the official olmOCR model weights.