# How to Troubleshoot Common olmOCR Errors: A Complete Diagnostic Guide

> Troubleshoot common olmOCR errors related to invalid data, network issues, or worker locks. Learn to diagnose and resolve these problems with this complete guide.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-08

---

**Most olmOCR failures stem from invalid input data (`ValidationError`), network connectivity issues (`ConnectionError`), or stale worker locks in the distributed queue, all of which can be diagnosed by inspecting the specific component logs in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py), and [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py).**

olmOCR is a distributed PDF-to-text pipeline developed by Allen Institute for AI (allenai/olmocr) that coordinates multiple subsystems including a work-queue manager, a vLLM inference engine, and S3 storage utilities. Because the system relies on asynchronous task processing across these components, errors can surface at the data validation layer, the network layer, or the filesystem layer. Understanding how to troubleshoot common olmOCR errors requires tracing the failure to its origin in the specific module responsible for work distribution, model inference, or storage access.

## 1. ValidationError: Malformed Input Data and Test Definitions

The `ValidationError` exception originates in [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py) when PDF filenames, test IDs, or required fields are empty or malformed.

When running the benchmark suite, this error typically appears at lines 106-112 in [`tests.py`](https://github.com/allenai/olmocr/blob/main/tests.py), where the code validates that every test object contains a non-empty `id`, `type`, and type-specific fields such as `text`, `before`, `after`, or `math`. The same exception type appears in PII tagging pipelines (e.g., [`scripts/pii/tagging_pipeline.py`](https://github.com/allenai/olmocr/blob/main/scripts/pii/tagging_pipeline.py) lines 28-30) when request payloads fail Pydantic model validation.

To troubleshoot `ValidationError`:

1. **Inspect the exception message** to identify which field is empty (e.g., "PDF filename cannot be empty").
2. **Validate your test JSON** against the schema in [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py) before execution.
3. **Add input guards** to prevent empty values from reaching the pipeline:

```python

# Example guard for PDF paths

if not pdf_path:
    raise ValueError("pdf_path must be a non-empty string")

```

4. **Re-run the bench suite** to confirm the fix:

```bash
pytest -k bench

```

## 2. ConnectionError and TimeoutError: vLLM Server Connectivity

Network failures occur when the inference engine cannot reach the vLLM server. In [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py), the low-level HTTP helper `apost()` raises `ConnectionError` at lines 22-27 if the TCP socket cannot be opened or the server returns no response.

If the vLLM server emits fatal sampler errors (such as a corrupted model checkpoint), the `vllm_server_task()` function logs "Cannot continue, sampling errors detected, model is probably corrupt" and aborts with `sys.exit(1)` at lines 62-64.

To diagnose and fix connection issues:

- **Verify server reachability** by running `curl http://HOST:PORT/models` to confirm the endpoint is listening.
- **Check server logs** for OOM or sampler errors indicating insufficient GPU memory or a corrupted checkpoint.
- **Increase the server-ready timeout** using `--max_server_ready_timeout` if the model requires extended warm-up time.
- **Verify API authentication** by setting `--api_key` or the `OPENAI_API_KEY` environment variable, as missing keys raise `ValueError` in PII pipelines like [`scripts/pii/autoscan_dolmadocs.py`](https://github.com/allenai/olmocr/blob/main/scripts/pii/autoscan_dolmadocs.py) (line 555).
- **Adjust retry behavior** by modifying `MAX_BACKOFF_ATTEMPTS` in [`pipeline.py`](https://github.com/allenai/olmocr/blob/main/pipeline.py) (line 64) to increase tolerance for transient failures.

For explicit retry handling with exponential back-off, use the built-in `try_single_page_with_backoff` function:

```python
import asyncio
from olmocr.pipeline import try_single_page_with_backoff

async def robust_page_request(args, pdf_path, page):
    # Retries up to 10 times with exponential back-off

    return await try_single_page_with_backoff(
        args, pdf_path, pdf_path, page, attempt=0, rotation=0
    )

# Usage inside an async worker

result = await robust_page_request(args, src_path, 1)

```

## 3. Worker Lock Staleness: "Lock Already Taken" Issues

In [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py), the `get_work()` method checks `backend.is_worker_lock_taken()` at lines 105-108 to skip items locked by other workers. If a worker process dies without releasing its lock, subsequent workers print "Work item … is locked by another worker, skipping" indefinitely.

To resolve worker lock staleness:

1. **Inspect lock timestamps** in the local backend by checking files under `worker_locks/worker_<hash>.lock` with `stat -c %y`.
2. **Increase the timeout** using `--worker_lock_timeout_secs` if your cluster experiences preemptions longer than the default 30 minutes.
3. **Clean up stale locks** manually:

```bash
find $WORKSPACE/worker_locks -type f -mmin +1800 -delete

```

This removes lock files older than 30 minutes, allowing the next worker to acquire them.

## 4. Model Download and Cache Failures

The `download_model()` function in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 56-63) attempts to download models from S3, local directories, or Hugging Face. Failures raise generic `Exception` types that propagate as pipeline aborts with the message "Could not download model".

Troubleshooting steps:

1. **Verify the model identifier** (`--model`) points to an accessible public HF repository or reachable S3 bucket.
2. **Clear the local cache** at `~/.cache/olmocr/model` to force a fresh download and eliminate corruption.
3. **Validate S3 credentials** if using cloud storage, ensuring the `workspace_s3` client has appropriate permissions.
4. **Increase context length** using `--max_model_len` if the default truncates large models.
5. **Check network connectivity** to `huggingface.co` or your S3 endpoint.

Safe model loading pattern:

```python
import asyncio
from olmocr.pipeline import download_model

async def safe_download(model_name):
    try:
        return await download_model(model_name)
    except Exception as exc:
        logger.error(f"Failed to acquire model {model_name}: {exc}")
        raise

```

## 5. S3-Related ClientError and Permission Issues

The [`olmocr/s3_utils.py`](https://github.com/allenai/olmocr/blob/main/olmocr/s3_utils.py) module raises `ClientError` for missing keys (`NoSuchKey`) or access denials (`PermissionDenied`), and `parse_s3_path()` raises `ValueError` for malformed URIs.

To fix S3 errors:

1. **Validate the S3 URI format** using `parse_s3_path()` to ensure it follows `s3://bucket/prefix/...`.
2. **Check IAM permissions** for `s3:GetObject`, `s3:PutObject`, and `s3:ListBucket` on the target prefixes.
3. **Confirm object existence** with `aws s3 ls s3://bucket/prefix/file.zst`.

Input validation example:

```python
from olmocr.s3_utils import parse_s3_path

def ensure_s3_uri(uri):
    bucket, key = parse_s3_path(uri)
    if not bucket or not key:
        raise ValueError(f"Invalid S3 URI: {uri}")

```

## 6. Filesystem Timeouts in LocalBackend

The `LocalBackend._run_with_timeout()` wrapper in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) (lines 54-63) aborts file operations after 120 seconds. When this timeout triggers, logs show "LocalBackend: TIMEOUT after 120 s on get_mtime(...)" and work items remain unmarked.

Resolution steps:

- **Check disk health** and I/O contention on the worker node.
- **Mount fast storage** (SSD) for the workspace directory instead of slow network filesystems.
- **Increase the timeout** by subclassing `LocalBackend` and setting `DEFAULT_FS_TIMEOUT` to a higher value if using NFS or similar slow storage.

## Robust Error Handling Patterns

For production deployments, implement these defensive patterns:

### Defensive Page Processing

```python
import asyncio
import logging
from olmocr.pipeline import try_single_page_with_backoff

logger = logging.getLogger(__name__)

async def process_page_safely(args, pdf_path, page):
    try:
        result = await try_single_page_with_backoff(
            args, pdf_path, pdf_path, page, attempt=0, rotation=0
        )
        if result is None:
            logger.warning(f"Page {page} produced no result – using fallback")
            return None
        return result
    except (ConnectionError, asyncio.TimeoutError) as e:
        logger.error(f"Network failure on page {page}: {e}")
        raise

```

### Stale Lock Cleanup Script

```bash
#!/usr/bin/env bash
WORKSPACE="/path/to/workspace"
LOCK_DIR="${WORKSPACE}/worker_locks"
find "$LOCK_DIR" -type f -mmin +45 -print -delete
echo "Stale locks cleared."

```

### Test JSON Validation

```python
import json
from pathlib import Path
from olmocr.bench.tests import ValidationError, TextPresenceTest

def validate_test_file(path: Path):
    with path.open() as f:
        data = json.load(f)
    try:
        TextPresenceTest(**data)
    except ValidationError as ve:
        raise ValueError(f"Invalid test definition in {path}: {ve}")

```

## Summary

- **ValidationError** in [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py) indicates malformed test definitions or empty required fields—validate JSON schemas before running benchmarks.
- **ConnectionError** in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) signals vLLM server unavailability—check server health, GPU memory, and API keys, and adjust `MAX_BACKOFF_ATTEMPTS` for transient failures.
- **Worker lock staleness** in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) occurs when workers die without releasing locks—clean `worker_locks/` directories and tune `--worker_lock_timeout_secs`.
- **Model download failures** require clearing `~/.cache/olmocr/model` and verifying network access to Hugging Face or S3.
- **S3 ClientError** exceptions require valid URI formatting (verified via `parse_s3_path()`) and proper IAM permissions.
- **Filesystem timeouts** in `LocalBackend` indicate slow storage—use SSDs or increase `DEFAULT_FS_TIMEOUT`.

## Frequently Asked Questions

### How do I fix ValidationError when running olmOCR benchmarks?

`ValidationError` occurs in [`olmocr/bench/tests.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/tests.py) (lines 106-112) when test JSON files contain empty `id` fields, missing `type` specifications, or invalid test parameters. To fix this, ensure every test object includes a non-empty `id`, a valid `type` (such as `text_presence` or `table_test`), and all required fields for that test type. Validate your JSON against the Pydantic models in [`tests.py`](https://github.com/allenai/olmocr/blob/main/tests.py) before execution.

### Why is my olmOCR worker skipping work items with "lock already taken" messages?

This indicates stale worker locks in the `LocalBackend` lock directory. The `get_work()` method in [`olmocr/work_queue.py`](https://github.com/allenai/olmocr/blob/main/olmocr/work_queue.py) (lines 105-108) skips items where `is_worker_lock_taken()` returns true based on file timestamps. If a previous worker died without releasing its lock, delete stale lock files from `worker_locks/` that are older than your `--worker_lock_timeout_secs` setting, or increase the timeout if your cluster experiences long preemptions.

### How do I resolve ConnectionError to the vLLM server in olmOCR?

`ConnectionError` originates in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) (lines 22-27) when the `apost()` helper cannot establish a TCP connection. Verify the server is running with `curl http://HOST:PORT/models`, check that GPU memory is sufficient (server logs will show OOM errors), and ensure you have set the correct API key via `--api_key` or `OPENAI_API_KEY`. If the model is large, increase `--max_server_ready_timeout` to accommodate slow warm-up times.

### What causes S3 ClientError in olmOCR and how do I fix it?

S3 errors stem from [`olmocr/s3_utils.py`](https://github.com/allenai/olmocr/blob/main/olmocr/s3_utils.py) when `download_zstd_csv` or `upload_zstd_csv` encounter missing keys or permission denials, or when `parse_s3_path()` receives malformed URIs. Ensure your S3 URIs follow the `s3://bucket/prefix` format, verify the bucket and key exist using `aws s3 ls`, and confirm your AWS credentials have `s3:GetObject`, `s3:PutObject`, and `s3:ListBucket` permissions for the target prefixes.