How to Install olmOCR for Local GPU Inference: Complete Setup Guide

Install olmOCR for local GPU inference by creating a Python 3.11 environment, installing system dependencies (poppler-utils and fonts), then running pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128 to deploy the Vision-Language Model pipeline.

The allenai/olmocr repository provides a production-ready pipeline for converting PDFs and images into clean Markdown using a local Vision-Language Model (VLM). When you install olmOCR for local GPU inference, you set up a complete stack that includes the olmocr[gpu] package, a vLLM server to serve the model, and PDF rendering utilities that prepare documents for processing. Understanding the architecture—from pipeline.py orchestrating the workflow to renderpdf.py handling image conversion—ensures you can troubleshoot and optimize your deployment.

Prerequisites and System Requirements

Hardware Requirements

olmOCR targets NVIDIA GPUs with CUDA 12.8 support. You need a compatible NVIDIA driver installed on your system before proceeding with the installation. The inference engine leverages vLLM to serve the model locally, which requires sufficient VRAM to load the Vision-Language Model weights and process batches of PDF pages.

Software Dependencies

Before installing the Python package, you must install system-level libraries required for PDF rendering. These dependencies allow olmocr/data/renderpdf.py to convert PDF pages into base64-encoded PNG images that the model consumes.

sudo apt-get update
sudo apt-get install -y poppler-utils \
    ttf-mscorefonts-installer msttcorefonts \
    fonts-crosextra-caladea fonts-crosextra-carlito \
    gsfonts lcdf-typetools

Step-by-Step Installation

Create a Python 3.11 Environment

olmOCR is tested against Python 3.11. Using a clean environment prevents dependency conflicts with other machine learning projects.

conda create -n olmocr python=3.11
conda activate olmocr

Install the GPU-Enabled Package

The olmocr[gpu] extra in pyproject.toml installs PyTorch 2.x, vLLM, and other heavy GPU dependencies required for local inference. You must specify the CUDA 12.8 index URL to ensure compatibility with the targeted CUDA version.

pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128

Optional: Flash-Infer for Accelerated Inference

For faster transformer-kernel execution on recent NVIDIA GPUs, install flash-infer. This optional dependency accelerates attention mechanisms within the VLM, reducing latency when processing large documents.

pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl

How the Local GPU Inference Pipeline Works

Understanding the architecture helps diagnose bottlenecks and optimize throughput.

pipeline.py functions as the central orchestrator in olmocr/pipeline.py. It coordinates three primary tasks: distributing work via work_queue.py (which implements LocalBackend and S3Backend), spawning a vLLM server through the vllm_server_task function (lines 807-828), and post-processing outputs. When processing begins, renderpdf.py converts each PDF page into a base64 PNG via render_pdf_to_base64png, which the vLLM server then feeds to the model through an OpenAI-compatible /chat/completions endpoint. Finally, process_page (lines 181-236 in pipeline.py) handles filtering, rotation correction, and Markdown generation.

Running Your First PDF Conversion

Once installation is complete, you can immediately convert PDFs to Markdown. The CLI requires a workspace directory for intermediate files and results.

# Download a sample PDF for testing

curl -o sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf

# Run the conversion with GPU acceleration

olmocr ./localworkspace \
    --markdown \
    --pdfs sample.pdf

Output Locations

The command creates two primary outputs relative to your workspace directory:

• Dolma JSONL: ./localworkspace/results/*.jsonl contains structured JSON documents for downstream training pipelines.
• Markdown: ./localworkspace/markdown/<original-path>.md provides the human-readable text extraction.

The --markdown flag explicitly enables the Markdown writer alongside the default Dolma output format.

Summary

• Install method: Use pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128 inside a Python 3.11 environment to deploy the full GPU stack.
• System deps: Install poppler-utils and Microsoft/Core fonts before running olmOCR to ensure accurate PDF rendering.
• Architecture: The pipeline.py module coordinates work_queue.py for job distribution and renderpdf.py for image preparation, while vLLM serves the model locally.
• Performance: Add flash-infer for accelerated kernel execution on modern NVIDIA hardware.
• Execution: Run olmocr <workspace> --markdown --pdfs <file> to generate both Dolma JSONL and Markdown outputs.

Frequently Asked Questions

What NVIDIA GPU models are compatible with olmOCR?

olmOCR requires CUDA 12.8 support, which means you need an NVIDIA GPU with compute capability 7.0 or higher (Volta architecture and newer). The installation command targets PyTorch wheels built for CUDA 12.8, so ensure your driver supports this version. You can verify CUDA availability by running nvidia-smi before installation.

How do I verify that the vLLM server is running correctly?

The vLLM server starts automatically when you run the olmocr command, initiated by the vllm_server_task function in olmocr/pipeline.py. Check your process logs for messages indicating the server is listening on a local port (typically 8000 or similar). If the server fails to start, verify that your CUDA drivers are properly installed and that no other process is occupying the target port.

Can I run olmOCR on CPU-only machines?

While the olmocr[gpu] package is designed for NVIDIA GPU acceleration, the repository structure supports CPU fallback in theory. However, the performance would be impractically slow for document conversion tasks, and the official installation instructions emphasize GPU deployment. For CPU-only environments, consider using the API-based inference options documented in the repository rather than local installation.

Where does olmOCR store intermediate files during processing?

olmOCR stores temporary PDF renderings, processing metadata, and final outputs in the workspace directory you specify as the first argument to the CLI command. According to olmocr/pipeline.py, this includes base64 PNG images generated by render_pdf_to_base64png and JSONL results. The work_queue.py module manages file handling through LocalBackend for local filesystem operations or S3Backend for cloud storage if configured.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}