# Dependencies for AllenAI olmocr: Core and Optional Packages Explained

> Learn about the core and optional dependencies for AllenAI olmocr including pypdf pypdfium2 and boto3 Discover GPU inference benchmarking and training extras in pyproject.toml

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: getting-started
- Published: 2026-07-07

---

**The `olmocr` library declares its runtime dependencies—including `pypdf`, `pypdfium2`, and `boto3`—and optional extras for GPU inference, benchmarking, and training inside the [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml) file.**

AllenAI's `olmocr` is an open-source OCR pipeline designed for high-volume PDF processing and text extraction. To install and run the tool effectively, you must understand the dependencies for allenai/olmocr, which are strictly managed through the [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml) configuration at the repository root. This file separates mandatory runtime libraries from optional feature sets required for GPU acceleration, cloud integration, and model development.

## Core Runtime Dependencies

These packages are required for all installations and are defined in the `dependencies` array of [[`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml)](https://github.com/allenai/olmocr/blob/main/pyproject.toml#L22-L40):

- `cached-path` – Uniform handling of cached file locations
- `smart_open` – Transparent I/O for local and cloud storage
- `pypdf>=5.2.0` – PDF parsing and manipulation
- `pypdfium2` – High-performance PDF rendering via PDFium
- `cryptography` – Secure handling of keys and certificates
- `lingua-language-detector` – Automatic language detection for OCR text
- `Pillow` – Image processing (resizing, format conversion)
- `ftfy` – Fixes Unicode text encoding problems
- `bleach` – Sanitizing HTML output from OCR
- `markdown2` – Converting plain text to Markdown
- `markdownify` – Converting HTML back to Markdown
- `filelock` – Simple file-based locking for concurrent processes
- `orjson` – Fast JSON serialization/deserialization
- `requests` – HTTP client for remote resources
- `zstandard` – Efficient compression of large OCR payloads
- `boto3` – AWS S3 access (used by the S3 work-queue)
- `httpx` – Async HTTP client (used by the evaluation pipeline)

## Optional Dependency Groups

AllenAI organizes optional functionality into extras that can be installed using `pip install olmocr[extra]`. These groups are defined in the `optional-dependencies` sections of [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml).

### GPU Support

The `gpu` extra (lines 52-56) installs deep learning libraries required for running OCR with large language-vision models:

- `torch>=2.7.0`
- `transformers==4.57.3`
- `vllm==0.11.2`

### Beaker Integration

The `beaker` extra (line 58) adds `beaker-py` for integration with the Beaker experiment-management platform.

### Development Tools

The `dev` extra (lines 60-74) includes linting, testing, and documentation utilities such as `ruff`, `mypy`, `black`, `isort`, and `pytest`.

### Benchmarking Suite

The `bench` extra (lines 92-106) provides tools for evaluation scripts:

- `tinyhost`, `fuzzysearch`, `rapidfuzz`, `sequence_align`, `syntok`
- API clients: `openai`, `google-genai`, `anthropic`, `mistralai`
- Utilities: `playwright`, `lxml`, `flask`, `wordfreq`

### Training Utilities

The `train` extra (lines 108-118) supports model-training pipelines:

- `torch`, `torchvision`, `accelerate`, `trl`, `peft`
- Experiment tracking: `wandb`
- Configuration: `omegaconf`
- Augmentation: `augraphy`

### ELO Rating Computation

The `elo` extra (lines 120-124) installs `numpy`, `scipy`, `pandas`, and `matplotlib` for computing and visualizing ELO ratings during model comparison.

## Installation Examples

Install the core package for basic PDF processing:

```bash
pip install olmocr

```

Install with GPU support for accelerated inference:

```bash
pip install "olmocr[gpu]"

```

Install multiple extras for a complete development environment:

```bash
pip install "olmocr[gpu,beaker,dev]"

```

## How Dependencies Are Used in Source Code

The declared dependencies power specific modules throughout the codebase.

**S3 Work Queue (`boto3`)**

The `boto3` library enables AWS S3 integration in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py). The `S3WorkQueue` class manages distributed processing queues:

```python
from olmocr.s3_utils import S3WorkQueue

queue = S3WorkQueue(bucket="my-olmocr-bucket")
queue.enqueue({"doc_id": "1234", "pdf_path": "s3://.../doc.pdf"})

```

**Pipeline Execution (`pypdf`, `Pillow`)**

The main entry point in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) orchestrates PDF parsing and image processing using `pypdf` and `Pillow`:

```python
from olmocr.pipeline import cli_main

# Run OCR pipeline programmatically

cli_main(["--input", "sample.pdf", "--output", "out.json"])

```

**Data Preprocessing**

The training data builder in [`olmocr/data/buildsilver.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/buildsilver.py) relies on core dependencies including `pypdfium2` for rendering and `ftfy` for text cleaning during dataset construction.

## Summary

- **Core dependencies** for allenai/olmocr are defined in [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml) lines 22-40 and include `pypdf`, `pypdfium2`, `boto3`, and `Pillow` for essential PDF and image handling.
- **Optional extras** provide GPU acceleration (`torch`, `vllm`), cloud platform integration (`beaker-py`), and development tools (`ruff`, `pytest`).
- Install specific feature sets using bracket notation: `pip install "olmocr[gpu,train]"`.
- Source files like [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) and [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) directly consume these dependencies for S3 operations and OCR processing.

## Frequently Asked Questions

### How do I install olmocr with GPU support for large language-vision models?

Use the `gpu` extra when installing. According to the [`pyproject.toml`](https://github.com/allenai/olmocr/blob/main/pyproject.toml) specification (lines 52-56), this installs `torch>=2.7.0`, `transformers==4.57.3`, and `vllm==0.11.2`. Run:

```bash
pip install "olmocr[gpu]"

```

### Why does olmocr require both pypdf and pypdfium2?

`pypdf>=5.2.0` handles PDF parsing and structural manipulation, while `pypdfium2` provides high-performance PDF rendering via the PDFium engine. This dual approach allows `olmocr` to extract text metadata and generate high-quality image renders for vision-language model processing.

### Can I use the S3 work queue without installing GPU dependencies?

Yes. The `boto3` dependency required for `S3WorkQueue` in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) is part of the core runtime requirements. You can install the base package with `pip install olmocr` and immediately use `S3WorkQueue` for distributed cloud processing without the `gpu` extra.

### What dependencies are needed to run the benchmarking scripts?

The `bench` extra (lines 92-106) installs all required packages including `rapidfuzz`, `syntok`, `playwright`, and API clients for `openai`, `anthropic`, and `google-genai`. Install these tools using:

```bash
pip install "olmocr[bench]"

```