Guided Decoding in olmOCR: Enforcing Strict YAML Output Validation for LLM Inference

Guided decoding is an optional inference mode that uses a regular expression constraint to force the language model to emit strictly formatted YAML front matter, eliminating malformed outputs before they reach downstream parsers.

The allenai/olmocr repository implements guided decoding to constrain language model outputs during PDF-to-text conversion. This technique ensures that generated metadata adheres to a strict YAML schema, preventing parsing errors in downstream training pipelines while maintaining high-throughput document processing.

What Is Guided Decoding?

Guided decoding is a regex-based output constraint mechanism integrated into the olmOCR inference pipeline. When activated via the --guided_decoding flag, it instructs the underlying language model—typically served via VLLM—to generate text that matches a predefined pattern before completing the response.

Technical Implementation in pipeline.py

The core logic resides in olmocr/pipeline.py. When the flag is present, the pipeline injects a guided_regex entry into the request query dictionary at lines 179-182:

if args.guided_decoding:
    query["guided_regex"] = (
        r"---\nprimary_language: (?:[a-z]{2}|null)\n"
        r"is_rotation_valid: (?:True|False|true|false)\n"
        r"rotation_correction: (?:0|90|180|270)\n"
        r"is_table: (?:True|False|true|false)\n"
        r"is_diagram: (?:True|False|true|false)\n"
        r"(?:---|---\n[\s\S]+)"
    )

This regular expression defines a YAML front-matter block delimited by --- markers and containing specific metadata fields about the processed document page.

Schema Structure

The enforced schema captures critical document properties:

• primary_language: Two-letter language code or null
• is_rotation_valid: Boolean indicating orientation correctness
• rotation_correction: Rotation angle (0, 90, 180, or 270 degrees)
• is_table: Boolean flag for tabular content detection
• is_diagram: Boolean flag for diagram detection

How Guided Decoding Improves YAML Validation

Guided decoding enhances output reliability through three distinct mechanisms that operate during the generation phase rather than during post-processing.

Schema Enforcement at Generation Time

Unlike post-hoc validation that rejects malformed outputs after generation, guided decoding prevents invalid structures from being produced entirely. The VLLM inference engine uses the provided regex to mask token probabilities, ensuring the model can only select tokens that advance the pattern match. This constraint-based generation dramatically reduces malformed or missing keys in the YAML block.

Early Detection of Format Errors

When the model violates the regex constraint—either by omitting required fields or introducing invalid syntax—the inference server returns a finish_reason other than stop. The pipeline explicitly checks this condition in olmocr/pipeline.py at lines 205-207:

if base_response_data["choices"][0]["finish_reason"] != "stop":
    is_valid = False

This early detection prevents downstream components from attempting to parse ambiguous or incomplete YAML, marking the result as invalid immediately and triggering retry logic or fallback extraction.

Consistent Downstream Parsing

The FrontMatterParser class in olmocr/train/front_matter.py expects a well-formed YAML header separated from body text by --- delimiters. Because guided decoding guarantees this structure syntactically, the parser can reliably split front matter from content without defensive programming against malformed inputs. This contract between generation and consumption layers streamlines the training data preparation workflow.

Implementing Guided Decoding in the Pipeline

Activation requires minimal configuration changes and integrates seamlessly with the existing olmOCR processing workflow defined in the try_single_page function.

Command Line Activation

The feature is exposed through the CLI argument parser defined at line 1231 of olmocr/pipeline.py. Enable guided decoding by passing the --guided_decoding flag:

python -m olmocr.pipeline \
    workspace /path/to/workspace \
    --model allenai/olmOCR-2-7B-1025-FP8 \
    --guided_decoding

This flag defaults to False, as confirmed by unit tests in tests/test_pipeline.py at line 199.

Request Payload Construction

Within the try_single_page async function, the pipeline constructs the query payload and conditionally appends the regex constraint:

query = await build_page_query(
    pdf_local_path,
    page_num,
    args.target_longest_image_dim,
    image_rotation=rotation,
    model_name=args.model,
)
query["temperature"] = temperature

if args.guided_decoding:
    query["guided_regex"] = (
        r"---\nprimary_language: (?:[a-z]{2}|null)\n"
        r"is_rotation_valid: (?:True|False|true|false)\n"
        r"rotation_correction: (?:0|90|180|270)\n"
        r"is_table: (?:True|False|true|false)\n"
        r"is_diagram: (?:True|False|true|false)\n"
        r"(?:---|---\n[\s\S]+)"
    )

Server-Side Constraint Handling

The payload is transmitted to a VLLM-compatible inference server. The server applies the regex as a guided generation constraint, sampling only tokens that maintain regex compliance. If the model cannot complete a valid match within the token limit, it returns an alternative finish reason, triggering the pipeline's validation failure handling and potential fallback to pdftotext extraction.

Practical Code Examples

Expected Model Output

When guided decoding succeeds, the model produces output matching this exact structure:

---
primary_language: en
is_rotation_valid: true
rotation_correction: 0
is_table: false
is_diagram: false
---
Here is the extracted natural text from the document...

Benchmark Runner Configuration

The benchmark runner in bench/runners/run_olmocr_pipeline.py at line 29 demonstrates programmatic toggling of this feature for evaluation purposes, allowing researchers to compare extraction quality with and without schema constraints.

Summary

• Guided decoding uses a regex constraint in olmocr/pipeline.py to enforce strict YAML front-matter formatting during language model inference.
• The technique prevents malformed outputs by constraining token generation at the inference server level rather than validating post-generation.
• Violations are detected via the finish_reason field at lines 205-207, allowing immediate invalidation of non-compliant responses.
• Downstream parsers in olmocr/train/front_matter.py benefit from guaranteed structural consistency, eliminating defensive parsing logic.
• Activation requires only the --guided_decoding CLI flag, with the regex schema defined at lines 179-182 of the pipeline module.

Frequently Asked Questions

What inference servers support guided decoding in olmOCR?

The pipeline is designed for VLLM-compatible inference servers that implement the guided_regex parameter in their completion APIs. When present in the request payload, compliant servers constrain token sampling to match the provided regular expression pattern before returning a response.

Does guided decoding impact processing speed?

Yes, constrained generation incurs minor overhead due to regex matching during token sampling. However, this cost is typically offset by reduced retry rates and elimination of downstream parsing failures that would otherwise require expensive reprocessing or manual intervention.

Can I modify the guided regex to add custom fields?

While the current implementation in olmocr/pipeline.py uses a hardcoded regex for the standard metadata schema, advanced users can modify the pattern at lines 179-182 to include additional YAML fields. Any modifications must remain compatible with the FrontMatterParser expectations in the training pipeline to avoid breaking downstream consumption.

How does the pipeline handle regex constraint violations?

When the model fails to satisfy the regex—indicated by a finish_reason other than stop in the response—the pipeline marks the page as invalid (is_valid = False) at lines 205-207 of olmocr/pipeline.py. Depending on configuration, the system either retries the extraction with adjusted parameters or falls back to deterministic pdftotext extraction for that specific page.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}