# How to Install olmOCR for Local GPU Inference: Complete Setup Guide

> Install olmOCR for local GPU inference with this setup guide. Learn to create a Python environment, add system dependencies, and run pip install olmocr with cu128 for efficient Vision-Language Model deployment.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-02

---

**Install olmOCR for local GPU inference by creating a Python 3.11 environment, installing system dependencies (poppler-utils and fonts), then running `pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128` to deploy the Vision-Language Model pipeline.**

The `allenai/olmocr` repository provides a production-ready pipeline for converting PDFs and images into clean Markdown using a local Vision-Language Model (VLM). When you install olmOCR for local GPU inference, you set up a complete stack that includes the `olmocr[gpu]` package, a vLLM server to serve the model, and PDF rendering utilities that prepare documents for processing. Understanding the architecture—from [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) orchestrating the workflow to [`renderpdf.py`](https://github.com/allenai/olmocr/blob/main/renderpdf.py) handling image conversion—ensures you can troubleshoot and optimize your deployment.

## Prerequisites and System Requirements

### Hardware Requirements

olmOCR targets **NVIDIA GPUs with CUDA 12.8** support. You need a compatible NVIDIA driver installed on your system before proceeding with the installation. The inference engine leverages vLLM to serve the model locally, which requires sufficient VRAM to load the Vision-Language Model weights and process batches of PDF pages.

### Software Dependencies

Before installing the Python package, you must install system-level libraries required for PDF rendering. These dependencies allow [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py) to convert PDF pages into base64-encoded PNG images that the model consumes.

```bash
sudo apt-get update
sudo apt-get install -y poppler-utils \
    ttf-mscorefonts-installer msttcorefonts \
    fonts-crosextra-caladea fonts-crosextra-carlito \
    gsfonts lcdf-typetools

```

## Step-by-Step Installation

### Create a Python 3.11 Environment

olmOCR is tested against Python 3.11. Using a clean environment prevents dependency conflicts with other machine learning projects.

```bash
conda create -n olmocr python=3.11
conda activate olmocr

```

### Install the GPU-Enabled Package

The `olmocr[gpu]` extra in [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml) installs PyTorch 2.x, vLLM, and other heavy GPU dependencies required for local inference. You must specify the CUDA 12.8 index URL to ensure compatibility with the targeted CUDA version.

```bash
pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128

```

### Optional: Flash-Infer for Accelerated Inference

For faster transformer-kernel execution on recent NVIDIA GPUs, install **flash-infer**. This optional dependency accelerates attention mechanisms within the VLM, reducing latency when processing large documents.

```bash
pip install https://download.pytorch.org/whl/cu128/flashinfer/flashinfer_python-0.2.5%2Bcu128torch2.7-cp38-abi3-linux_x86_64.whl

```

## How the Local GPU Inference Pipeline Works

Understanding the architecture helps diagnose bottlenecks and optimize throughput.

**[`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)** functions as the central orchestrator in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). It coordinates three primary tasks: distributing work via [`work_queue.py`](https://github.com/allenai/olmocr/blob/main/work_queue.py) (which implements `LocalBackend` and `S3Backend`), spawning a vLLM server through the `vllm_server_task` function (lines 807-828), and post-processing outputs. When processing begins, [`renderpdf.py`](https://github.com/allenai/olmocr/blob/main/renderpdf.py) converts each PDF page into a base64 PNG via `render_pdf_to_base64png`, which the vLLM server then feeds to the model through an OpenAI-compatible `/chat/completions` endpoint. Finally, `process_page` (lines 181-236 in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py)) handles filtering, rotation correction, and Markdown generation.

## Running Your First PDF Conversion

Once installation is complete, you can immediately convert PDFs to Markdown. The CLI requires a workspace directory for intermediate files and results.

```bash

# Download a sample PDF for testing

curl -o sample.pdf https://olmocr.allenai.org/papers/olmocr_3pg_sample.pdf

# Run the conversion with GPU acceleration

olmocr ./localworkspace \
    --markdown \
    --pdfs sample.pdf

```

**Output Locations**

The command creates two primary outputs relative to your workspace directory:

- **Dolma JSONL**: `./localworkspace/results/*.jsonl` contains structured JSON documents for downstream training pipelines.
- **Markdown**: `./localworkspace/markdown/<original-path>.md` provides the human-readable text extraction.

The `--markdown` flag explicitly enables the Markdown writer alongside the default Dolma output format.

## Summary

- **Install method**: Use `pip install "olmocr[gpu]" --extra-index-url https://download.pytorch.org/whl/cu128` inside a Python 3.11 environment to deploy the full GPU stack.
- **System deps**: Install `poppler-utils` and Microsoft/Core fonts before running olmOCR to ensure accurate PDF rendering.
- **Architecture**: The [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) module coordinates [`work_queue.py`](https://github.com/allenai/olmocr/blob/main/work_queue.py) for job distribution and [`renderpdf.py`](https://github.com/allenai/olmocr/blob/main/renderpdf.py) for image preparation, while vLLM serves the model locally.
- **Performance**: Add flash-infer for accelerated kernel execution on modern NVIDIA hardware.
- **Execution**: Run `olmocr <workspace> --markdown --pdfs <file>` to generate both Dolma JSONL and Markdown outputs.

## Frequently Asked Questions

### What NVIDIA GPU models are compatible with olmOCR?

olmOCR requires CUDA 12.8 support, which means you need an NVIDIA GPU with compute capability 7.0 or higher (Volta architecture and newer). The installation command targets PyTorch wheels built for CUDA 12.8, so ensure your driver supports this version. You can verify CUDA availability by running `nvidia-smi` before installation.

### How do I verify that the vLLM server is running correctly?

The vLLM server starts automatically when you run the `olmocr` command, initiated by the `vllm_server_task` function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). Check your process logs for messages indicating the server is listening on a local port (typically 8000 or similar). If the server fails to start, verify that your CUDA drivers are properly installed and that no other process is occupying the target port.

### Can I run olmOCR on CPU-only machines?

While the `olmocr[gpu]` package is designed for NVIDIA GPU acceleration, the repository structure supports CPU fallback in theory. However, the performance would be impractically slow for document conversion tasks, and the official installation instructions emphasize GPU deployment. For CPU-only environments, consider using the API-based inference options documented in the repository rather than local installation.

### Where does olmOCR store intermediate files during processing?

olmOCR stores temporary PDF renderings, processing metadata, and final outputs in the workspace directory you specify as the first argument to the CLI command. According to [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), this includes base64 PNG images generated by `render_pdf_to_base64png` and JSONL results. The [`work_queue.py`](https://github.com/allenai/olmocr/blob/main/work_queue.py) module manages file handling through `LocalBackend` for local filesystem operations or `S3Backend` for cloud storage if configured.