How to Use olmOCR with a Remote Inference Server: A Complete Configuration Guide

You can scale olmOCR to millions of PDFs by delegating vision-language model inference to any server implementing the OpenAI Chat Completions API using the --server and --api_key flags, while the pipeline automatically handles PDF rendering, prompt construction, and error recovery.

The allenai/olmocr repository provides a production-ready pipeline that separates document processing from model inference. By pointing the pipeline at a remote inference server—such as a vLLM instance or managed API provider—you can distribute GPU-intensive work across specialized hardware while the orchestration logic handles PDF parsing, image encoding, and result assembly.

How the Remote Inference Architecture Works

When you configure olmOCR with a remote server, the pipeline executes an eight-step workflow that transforms raw PDFs into structured Dolma documents.

PDF Rendering and Encoding

First, render_pdf_to_base64png in olmocr/data/renderpdf.py converts each PDF page into a base64-encoded PNG image. This function handles DPI scaling, color space conversion, and memory-efficient streaming to support documents with thousands of pages.

Prompt Construction

Next, build_page_query (located in olmocr/pipeline.py) assembles the chat completion payload. It injects the base64 image alongside the system prompt generated by build_no_anchoring_v4_yaml_prompt from olmocr/prompts/prompts.py. The resulting JSON payload follows the OpenAI vision format, with the image URL set to data:image/png;base64,{encoded_string}.

HTTP Transport and Authentication

The apost async function in olmocr/pipeline.py opens a low-level TCP connection to the remote server. When you provide --api_key, the function adds an Authorization: Bearer <key> header to every POST request. The target URL must end with /chat/completions, and the pipeline expects a JSON response containing usage token counts and choices[0].message.content.

Response Handling and Fallbacks

try_single_page parses the model response, extracts the document structure into a PageResponse object, and records metrics via MetricsKeeper. If the model returns a rotation error, the pipeline automatically retries with corrected angles. When servers return persistent errors or context-length exceeded messages, the system falls back to make_fallback_result, which uses pdftotext for pure text extraction.

Document Assembly and Storage

Finally, build_dolma_document merges all page results into a single JSONL record with per-page metadata (detected language, rotation angles, table structures). The s3_utils module writes completed documents to your workspace path, whether local filesystem or S3.

Essential Configuration Flags

To connect olmOCR to a remote inference server, you need three key parameters:

• --server – The base URL of the remote endpoint (e.g., http://localhost:8000 or https://api.deepinfra.com/v1/openai). The pipeline appends /chat/completions automatically.
• --api_key – Optional bearer token for authenticated endpoints. The pipeline includes this as Authorization: Bearer <key> in HTTP headers.
• --max_concurrent_requests – Limits parallel HTTP connections to prevent overwhelming rate-limited services. Adjust this based on your remote server's capacity.
• --workers – Local process pool size for PDF preprocessing tasks (rendering and prompt building), which run independently of the inference requests.

End-to-End Configuration Examples

Connecting to a Public vLLM Server

The simplest deployment points olmOCR at an existing vLLM instance hosting the allenai/olmOCR-2-7B-1025-FP8 model:

python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --server http://my-vllm-instance.example.com:8000 \
  --max_concurrent_requests 300 \
  --workers 20

This configuration streams PDFs from S3, sends each page to the remote server, and writes JSONL results back to s3://my-bucket/olmocr-workspace/results/.

Authenticated Providers (DeepInfra)

For commercial APIs requiring authentication, export your key and reference it via --api_key:

export DEEPINFRA_API_KEY=sk-xxxxxxx

python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model deepinfra/olmocr-2b \
  --server https://api.deepinfra.com/v1/openai \
  --api_key $DEEPINFRA_API_KEY \
  --max_concurrent_requests 100 \
  --workers 10

The pipeline automatically injects the bearer token into every request to https://api.deepinfra.com/v1/openai/chat/completions.

Local vLLM Server with Distributed Workers

You can host the vLLM server separately and connect multiple olmOCR workers to it:

1. Start the inference server (one terminal):

python -m olmocr.pipeline \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --port 8000 \
  --launch_vllm

2. Run processing workers (other terminals):

python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --server http://127.0.0.1:8000 \
  --workers 40 \
  --max_concurrent_requests 800

The --launch_vllm flag triggers vllm_server_task in olmocr/pipeline.py, which initializes the model with optimal tensor parallelism settings.

Generating Markdown Output

To produce human-readable .md files alongside the JSONL records:

python -m olmocr.pipeline s3://my-bucket/olmocr-workspace \
  --pdfs s3://my-bucket/pdfs/*.pdf \
  --model allenai/olmOCR-2-7B-1025-FP8 \
  --server http://my-vllm-instance.example.com:8000 \
  --markdown \
  --workers 15

When --markdown is set, the pipeline writes cleaned text to <workspace>/markdown/ using get_markdown_path for filename generation.

Reusing Pipeline Components in Python

If you need to call the remote server outside the full pipeline context, import the apost helper directly:

import asyncio
import json
from olmocr.pipeline import apost

async def remote_olmocr(query: dict, server: str, api_key: str | None = None):
    """
    Send a single page query to the remote inference server.
    
    Args:
        query: Dictionary containing the OpenAI-compatible chat completion payload
        server: Base URL of the inference endpoint
        api_key: Optional bearer token for authentication
    """
    url = f"{server.rstrip('/')}/chat/completions"
    status, body = await apost(url, json_data=query, api_key=api_key)
    
    if status != 200:
        raise RuntimeError(f"Server returned HTTP {status}: {body}")
    
    result = json.loads(body)
    return result["choices"][0]["message"]["content"]

# Usage example

async def main():
    query = {
        "model": "allenai/olmOCR-2-7B-1025-FP8",
        "messages": [
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "Convert this page to markdown"},
                    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}}
                ]
            }
        ]
    }
    content = await remote_olmocr(query, "http://localhost:8000")
    print(content)

asyncio.run(main())

This low-level approach uses the same apost function that the full pipeline relies on for connection pooling, timeout handling, and retry logic.

Summary

• Use the --server flag to point olmOCR at any OpenAI-compatible Chat Completions endpoint, including vLLM and commercial providers.
• Authenticate private endpoints by passing --api_key, which the pipeline converts to Authorization: Bearer headers in olmocr/pipeline.py.
• The pipeline automatically encodes PDF pages via render_pdf_to_base64png and constructs vision-language prompts through build_page_query.
• Automatic backoff and rotation correction are handled by try_single_page, with fallback to pdftotext if the remote server fails persistently.
• Scale horizontally by running one vLLM server with --launch_vllm and connecting multiple olmOCR worker processes via --server.

Frequently Asked Questions

What API format must the remote inference server implement?

The remote server must expose an OpenAI-compatible Chat Completions endpoint at /chat/completions. It should accept JSON POST requests with model, messages, and optional max_tokens parameters, returning responses containing choices[0].message.content and usage metadata. vLLM, TGI, and most commercial providers (DeepInfra, OpenAI, Together) support this format natively.

How does olmOCR handle authentication for private inference endpoints?

When you provide the --api_key flag, the apost function in olmocr/pipeline.py automatically adds an Authorization: Bearer <token> header to every HTTP request. This supports standard JWT authentication used by private vLLM deployments and cloud API providers without modifying the underlying code.

What happens if the remote server returns errors or timeouts?

The pipeline implements exponential backoff in try_single_page_with_backoff for transient network failures. For specific errors like rotation mismatches, it automatically retries with corrected angles. If the server returns persistent errors (context length exceeded, repeated 5xx codes), the pipeline triggers make_fallback_result to extract text using pdftotext, ensuring the workflow completes even when inference fails.

Can I use olmOCR with standard cloud LLMs like GPT-4 or Claude?

While technically possible if the provider supports image inputs via the Chat Completions API, olmOCR is optimized for the allenai/olmOCR-2-7B-1025-FP8 model architecture. Standard cloud LLMs may not follow the specific prompt templates defined in olmocr/prompts/prompts.py or return the structured YAML front-matter that try_single_page expects for document assembly. For best results, use a vLLM server hosting the official olmOCR model weights.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}