Dependencies for AllenAI olmocr: Core and Optional Packages Explained

The olmocr library declares its runtime dependencies—including pypdf, pypdfium2, and boto3—and optional extras for GPU inference, benchmarking, and training inside the pyproject.toml file.

AllenAI's olmocr is an open-source OCR pipeline designed for high-volume PDF processing and text extraction. To install and run the tool effectively, you must understand the dependencies for allenai/olmocr, which are strictly managed through the pyproject.toml configuration at the repository root. This file separates mandatory runtime libraries from optional feature sets required for GPU acceleration, cloud integration, and model development.

Core Runtime Dependencies

These packages are required for all installations and are defined in the dependencies array of [pyproject.toml](https://github.com/allenai/olmocr/blob/main/pyproject.toml#L22-L40):

• cached-path – Uniform handling of cached file locations
• smart_open – Transparent I/O for local and cloud storage
• pypdf>=5.2.0 – PDF parsing and manipulation
• pypdfium2 – High-performance PDF rendering via PDFium
• cryptography – Secure handling of keys and certificates
• lingua-language-detector – Automatic language detection for OCR text
• Pillow – Image processing (resizing, format conversion)
• ftfy – Fixes Unicode text encoding problems
• bleach – Sanitizing HTML output from OCR
• markdown2 – Converting plain text to Markdown
• markdownify – Converting HTML back to Markdown
• filelock – Simple file-based locking for concurrent processes
• orjson – Fast JSON serialization/deserialization
• requests – HTTP client for remote resources
• zstandard – Efficient compression of large OCR payloads
• boto3 – AWS S3 access (used by the S3 work-queue)
• httpx – Async HTTP client (used by the evaluation pipeline)

Optional Dependency Groups

AllenAI organizes optional functionality into extras that can be installed using pip install olmocr[extra]. These groups are defined in the optional-dependencies sections of pyproject.toml.

GPU Support

The gpu extra (lines 52-56) installs deep learning libraries required for running OCR with large language-vision models:

• torch>=2.7.0
• transformers==4.57.3
• vllm==0.11.2

Beaker Integration

The beaker extra (line 58) adds beaker-py for integration with the Beaker experiment-management platform.

Development Tools

The dev extra (lines 60-74) includes linting, testing, and documentation utilities such as ruff, mypy, black, isort, and pytest.

Benchmarking Suite

The bench extra (lines 92-106) provides tools for evaluation scripts:

• tinyhost, fuzzysearch, rapidfuzz, sequence_align, syntok
• API clients: openai, google-genai, anthropic, mistralai
• Utilities: playwright, lxml, flask, wordfreq

Training Utilities

The train extra (lines 108-118) supports model-training pipelines:

• torch, torchvision, accelerate, trl, peft
• Experiment tracking: wandb
• Configuration: omegaconf
• Augmentation: augraphy

ELO Rating Computation

The elo extra (lines 120-124) installs numpy, scipy, pandas, and matplotlib for computing and visualizing ELO ratings during model comparison.

Installation Examples

Install the core package for basic PDF processing:

pip install olmocr

Install with GPU support for accelerated inference:

pip install "olmocr[gpu]"

Install multiple extras for a complete development environment:

pip install "olmocr[gpu,beaker,dev]"

How Dependencies Are Used in Source Code

The declared dependencies power specific modules throughout the codebase.

S3 Work Queue (boto3)

The boto3 library enables AWS S3 integration in olmocr/work_queue.py. The S3WorkQueue class manages distributed processing queues:

from olmocr.s3_utils import S3WorkQueue

queue = S3WorkQueue(bucket="my-olmocr-bucket")
queue.enqueue({"doc_id": "1234", "pdf_path": "s3://.../doc.pdf"})

Pipeline Execution (pypdf, Pillow)

The main entry point in olmocr/pipeline.py orchestrates PDF parsing and image processing using pypdf and Pillow:

from olmocr.pipeline import cli_main

# Run OCR pipeline programmatically

cli_main(["--input", "sample.pdf", "--output", "out.json"])

Data Preprocessing

The training data builder in olmocr/data/buildsilver.py relies on core dependencies including pypdfium2 for rendering and ftfy for text cleaning during dataset construction.

Summary

• Core dependencies for allenai/olmocr are defined in pyproject.toml lines 22-40 and include pypdf, pypdfium2, boto3, and Pillow for essential PDF and image handling.
• Optional extras provide GPU acceleration (torch, vllm), cloud platform integration (beaker-py), and development tools (ruff, pytest).
• Install specific feature sets using bracket notation: pip install "olmocr[gpu,train]".
• Source files like olmocr/work_queue.py and olmocr/pipeline.py directly consume these dependencies for S3 operations and OCR processing.

Frequently Asked Questions

How do I install olmocr with GPU support for large language-vision models?

Use the gpu extra when installing. According to the pyproject.toml specification (lines 52-56), this installs torch>=2.7.0, transformers==4.57.3, and vllm==0.11.2. Run:

pip install "olmocr[gpu]"

Why does olmocr require both pypdf and pypdfium2?

pypdf>=5.2.0 handles PDF parsing and structural manipulation, while pypdfium2 provides high-performance PDF rendering via the PDFium engine. This dual approach allows olmocr to extract text metadata and generate high-quality image renders for vision-language model processing.

Can I use the S3 work queue without installing GPU dependencies?

Yes. The boto3 dependency required for S3WorkQueue in olmocr/work_queue.py is part of the core runtime requirements. You can install the base package with pip install olmocr and immediately use S3WorkQueue for distributed cloud processing without the gpu extra.

What dependencies are needed to run the benchmarking scripts?

The bench extra (lines 92-106) installs all required packages including rapidfuzz, syntok, playwright, and API clients for openai, anthropic, and google-genai. Install these tools using:

pip install "olmocr[bench]"

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}