# Guided Decoding in olmOCR: Enforcing Strict YAML Output Validation for LLM Inference

> Explore guided decoding in olmOCR and its powerful regex constraint for strict YAML output validation during LLM inference. Get perfectly formatted data every time.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: deep-dive
- Published: 2026-07-06

---

**Guided decoding is an optional inference mode that uses a regular expression constraint to force the language model to emit strictly formatted YAML front matter, eliminating malformed outputs before they reach downstream parsers.**

The allenai/olmocr repository implements guided decoding to constrain language model outputs during PDF-to-text conversion. This technique ensures that generated metadata adheres to a strict YAML schema, preventing parsing errors in downstream training pipelines while maintaining high-throughput document processing.

## What Is Guided Decoding?

Guided decoding is a regex-based output constraint mechanism integrated into the olmOCR inference pipeline. When activated via the `--guided_decoding` flag, it instructs the underlying language model—typically served via VLLM—to generate text that matches a predefined pattern before completing the response.

### Technical Implementation in pipeline.py

The core logic resides in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). When the flag is present, the pipeline injects a `guided_regex` entry into the request query dictionary at lines 179-182:

```python
if args.guided_decoding:
    query["guided_regex"] = (
        r"---\nprimary_language: (?:[a-z]{2}|null)\n"
        r"is_rotation_valid: (?:True|False|true|false)\n"
        r"rotation_correction: (?:0|90|180|270)\n"
        r"is_table: (?:True|False|true|false)\n"
        r"is_diagram: (?:True|False|true|false)\n"
        r"(?:---|---\n[\s\S]+)"
    )

```

This regular expression defines a YAML front-matter block delimited by `---` markers and containing specific metadata fields about the processed document page.

### Schema Structure

The enforced schema captures critical document properties:

- **primary_language**: Two-letter language code or null
- **is_rotation_valid**: Boolean indicating orientation correctness
- **rotation_correction**: Rotation angle (0, 90, 180, or 270 degrees)
- **is_table**: Boolean flag for tabular content detection
- **is_diagram**: Boolean flag for diagram detection

## How Guided Decoding Improves YAML Validation

Guided decoding enhances output reliability through three distinct mechanisms that operate during the generation phase rather than during post-processing.

### Schema Enforcement at Generation Time

Unlike post-hoc validation that rejects malformed outputs after generation, guided decoding prevents invalid structures from being produced entirely. The VLLM inference engine uses the provided regex to mask token probabilities, ensuring the model can only select tokens that advance the pattern match. This **constraint-based generation** dramatically reduces malformed or missing keys in the YAML block.

### Early Detection of Format Errors

When the model violates the regex constraint—either by omitting required fields or introducing invalid syntax—the inference server returns a `finish_reason` other than `stop`. The pipeline explicitly checks this condition in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) at lines 205-207:

```python
if base_response_data["choices"][0]["finish_reason"] != "stop":
    is_valid = False

```

This early detection prevents downstream components from attempting to parse ambiguous or incomplete YAML, marking the result as invalid immediately and triggering retry logic or fallback extraction.

### Consistent Downstream Parsing

The `FrontMatterParser` class in [`olmocr/train/front_matter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/front_matter.py) expects a well-formed YAML header separated from body text by `---` delimiters. Because guided decoding guarantees this structure syntactically, the parser can reliably split front matter from content without defensive programming against malformed inputs. This contract between generation and consumption layers streamlines the training data preparation workflow.

## Implementing Guided Decoding in the Pipeline

Activation requires minimal configuration changes and integrates seamlessly with the existing olmOCR processing workflow defined in the `try_single_page` function.

### Command Line Activation

The feature is exposed through the CLI argument parser defined at line 1231 of [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). Enable guided decoding by passing the `--guided_decoding` flag:

```bash
python -m olmocr.pipeline \
    workspace /path/to/workspace \
    --model allenai/olmOCR-2-7B-1025-FP8 \
    --guided_decoding

```

This flag defaults to `False`, as confirmed by unit tests in [`tests/test_pipeline.py`](https://github.com/allenai/olmocr/blob/main/tests/test_pipeline.py) at line 199.

### Request Payload Construction

Within the `try_single_page` async function, the pipeline constructs the query payload and conditionally appends the regex constraint:

```python
query = await build_page_query(
    pdf_local_path,
    page_num,
    args.target_longest_image_dim,
    image_rotation=rotation,
    model_name=args.model,
)
query["temperature"] = temperature

if args.guided_decoding:
    query["guided_regex"] = (
        r"---\nprimary_language: (?:[a-z]{2}|null)\n"
        r"is_rotation_valid: (?:True|False|true|false)\n"
        r"rotation_correction: (?:0|90|180|270)\n"
        r"is_table: (?:True|False|true|false)\n"
        r"is_diagram: (?:True|False|true|false)\n"
        r"(?:---|---\n[\s\S]+)"
    )

```

### Server-Side Constraint Handling

The payload is transmitted to a VLLM-compatible inference server. The server applies the regex as a **guided generation constraint**, sampling only tokens that maintain regex compliance. If the model cannot complete a valid match within the token limit, it returns an alternative finish reason, triggering the pipeline's validation failure handling and potential fallback to `pdftotext` extraction.

## Practical Code Examples

### Expected Model Output

When guided decoding succeeds, the model produces output matching this exact structure:

```yaml
---
primary_language: en
is_rotation_valid: true
rotation_correction: 0
is_table: false
is_diagram: false
---
Here is the extracted natural text from the document...

```

### Benchmark Runner Configuration

The benchmark runner in [`bench/runners/run_olmocr_pipeline.py`](https://github.com/allenai/olmocr/blob/main/bench/runners/run_olmocr_pipeline.py) at line 29 demonstrates programmatic toggling of this feature for evaluation purposes, allowing researchers to compare extraction quality with and without schema constraints.

## Summary

- Guided decoding uses a regex constraint in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) to enforce strict YAML front-matter formatting during language model inference.
- The technique prevents malformed outputs by constraining token generation at the inference server level rather than validating post-generation.
- Violations are detected via the `finish_reason` field at lines 205-207, allowing immediate invalidation of non-compliant responses.
- Downstream parsers in [`olmocr/train/front_matter.py`](https://github.com/allenai/olmocr/blob/main/olmocr/train/front_matter.py) benefit from guaranteed structural consistency, eliminating defensive parsing logic.
- Activation requires only the `--guided_decoding` CLI flag, with the regex schema defined at lines 179-182 of the pipeline module.

## Frequently Asked Questions

### What inference servers support guided decoding in olmOCR?

The pipeline is designed for VLLM-compatible inference servers that implement the `guided_regex` parameter in their completion APIs. When present in the request payload, compliant servers constrain token sampling to match the provided regular expression pattern before returning a response.

### Does guided decoding impact processing speed?

Yes, constrained generation incurs minor overhead due to regex matching during token sampling. However, this cost is typically offset by reduced retry rates and elimination of downstream parsing failures that would otherwise require expensive reprocessing or manual intervention.

### Can I modify the guided regex to add custom fields?

While the current implementation in [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py) uses a hardcoded regex for the standard metadata schema, advanced users can modify the pattern at lines 179-182 to include additional YAML fields. Any modifications must remain compatible with the `FrontMatterParser` expectations in the training pipeline to avoid breaking downstream consumption.

### How does the pipeline handle regex constraint violations?

When the model fails to satisfy the regex—indicated by a `finish_reason` other than `stop` in the response—the pipeline marks the page as invalid (`is_valid = False`) at lines 205-207 of [`olmocr/pipeline.py`](https://github.com/allenai/olmocr/blob/main/olmocr/pipeline.py). Depending on configuration, the system either retries the extraction with adjusted parameters or falls back to deterministic `pdftotext` extraction for that specific page.