Understanding the Temperature Retry Mechanism in OlmOCR: Handling Model Failures

OlmOCR uses a temperature-based retry mechanism that progressively increases the LLM's sampling temperature from 0.1 to 1.0 across eight attempts to escape repetitive generation patterns and recover from model failures.

The allenai/olmocr pipeline implements a sophisticated temperature retry mechanism to ensure robust OCR processing when large language models encounter generation errors or repetitive outputs. By gradually escalating the randomness of model sampling, this system provides multiple recovery paths before falling back to deterministic OCR engines.

How the Temperature Retry Mechanism Works

The mechanism relies on a predefined schedule in olmocr/pipeline.py that maps each retry attempt to a specific temperature value, balancing determinism with recovery potential.

The Temperature Schedule

The static list TEMPERATURE_BY_ATTEMPT defines the progression:

# olmocr/pipeline.py

TEMPERATURE_BY_ATTEMPT = [0.1, 0.1, 0.2, 0.3, 0.5, 0.8, 0.9, 1.0]

This configuration allows two initial attempts at very low temperature (0.1), followed by gradual escalation to maximum randomness (1.0) on the eighth attempt. Lower temperatures prioritize deterministic outputs for well-behaved pages, while higher temperatures inject diversity to break error loops.

Temperature Assignment Logic

The try_single_page function selects the appropriate temperature based on the current attempt counter:

# olmocr/pipeline.py

temp_idx = min(attempt, len(TEMPERATURE_BY_ATTEMPT) - 1)
temperature = TEMPERATURE_BY_ATTEMPT[temp_idx]
query["temperature"] = temperature

The code clamps the attempt index to prevent array overflows while ensuring each retry receives progressively higher temperature values in the LLM query payload sent to /chat/completions.

The Retry Loop Orchestration

The process_page function manages the retry lifecycle:

# olmocr/pipeline.py

MAX_RETRIES = args.max_page_retries
retry_attempts = list(range(1, MAX_RETRIES))

# First attempt (attempt 0) uses temperature 0.1

# Subsequent failures trigger retries with escalating temperatures

result = await try_single_page_with_backoff(..., attempt, cumulative_rotation)

The first attempt executes with the lowest temperature (0.1). Upon validation failure, the loop advances to the next attempt, automatically selecting higher temperatures according to the schedule until either validation passes or the list exhausts.

Handling Different Failure Types

The mechanism distinguishes between transient network issues and substantive model generation failures, applying different strategies for each.

HTTP and Content Validation Failures

When the remote server returns a non-200 status code, try_single_page returns None, triggering the temperature escalation sequence. For successful HTTP responses, the pipeline validates:

• total_tokens ≤ MODEL_MAX_CONTEXT
• finish_reason == "stop"

If either check fails, the system marks the result invalid and proceeds to the next temperature tier, allowing the model to generate alternative responses under different sampling conditions.

Connection Errors and Backoff Strategy

Network interruptions are handled separately through try_single_page_with_backoff, which wraps the core function with exponential backoff timing (10 seconds × 2^n). These backoff retries are independent of the temperature schedule—they wait for network recovery while maintaining the same temperature value, preserving the escalation strategy for generation failures only.

Fallback Strategy When Retries Exhaust

If all eight temperature retries fail to produce valid output, the pipeline invokes make_fallback_result to ensure document completion:

# olmocr/pipeline.py

return make_fallback_result(pdf_orig_path, pdf_local_path, page_num)

This function switches to the pdftotext OCR engine, guaranteeing that LLM failures never block the overall processing workflow. The fallback acts as a safety net when the temperature retry mechanism cannot resolve model-specific generation errors.

Practical Implementation Example

The following example demonstrates how to exercise the retry logic for a single page:

import asyncio
from olmocr.pipeline import try_single_page, try_single_page_with_backoff, TEMPERATURE_BY_ATTEMPT

async def demo_retry(args, pdf_path, page):
    for attempt in range(len(TEMPERATURE_BY_ATTEMPT)):
        result = await try_single_page_with_backoff(
            args,
            pdf_orig_path=pdf_path,
            pdf_local_path=pdf_path,
            page_num=page,
            attempt=attempt,
            rotation=0,
        )
        if result and result.is_valid and result.response.is_rotation_valid:
            print(f"Success on attempt {attempt} (temp={TEMPERATURE_BY_ATTEMPT[attempt]})")
            break
        else:
            print(f"Attempt {attempt} failed – retrying with higher temperature")
    else:
        print("All retries exhausted – fallback will be used")

Summary

• Progressive temperature escalation: The TEMPERATURE_BY_ATTEMPT array in olmocr/pipeline.py starts at 0.1 (near-deterministic) and scales to 1.0 (maximum randomness) across eight attempts.
• Separation of concerns: Network failures trigger exponential backoff without temperature changes, while generation failures trigger temperature increases via try_single_page.
• Deterministic fallback: The make_fallback_result function ensures processing completion via pdftotext when all LLM retries exhaust, preventing pipeline stalls.

Frequently Asked Questions

Why does OlmOCR use temperature escalation instead of fixed retries?

Fixing a low temperature can trap the model in repetitive error patterns when it encounters difficult layouts or ambiguous text. By gradually increasing temperature from 0.1 to 1.0, the system allows the LLM to explore alternative phrasings and escape local minima in the probability distribution without sacrificing determinism on easier pages.

How many retry attempts does OlmOCR allow by default?

The default configuration supports up to eight attempts, defined by the eight values in TEMPERATURE_BY_ATTEMPT. The MAX_RETRIES variable is set from args.max_page_retries, allowing customization while being clamped to the length of the temperature schedule to prevent index errors.

Does raising the temperature guarantee successful OCR?

No. Higher temperatures increase output diversity but do not guarantee validity. If all eight temperature levels fail validation checks—such as exceeding MODEL_MAX_CONTEXT or returning a non-stop finish reason—the system falls back to pdftotext to ensure the pipeline delivers a result rather than failing silently.

Where is the temperature retry mechanism implemented in the codebase?

The core logic resides in olmocr/pipeline.py, specifically within the TEMPERATURE_BY_ATTEMPT schedule (lines 84-87), the try_single_page function that assigns temperatures to queries, and the process_page function that orchestrates the retry loop and validation logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}