# How olmOCR Handles Multi-Column PDF Layouts and Reading Order Detection

> Discover how olmOCR tackles multi-column PDFs using a vision-first pipeline for accurate reading order detection and semantic HTML generation. Learn more today.

- Repository: [Ai2/olmocr](https://github.com/allenai/olmocr)
- Tags: how-to-guide
- Published: 2026-07-05

---

**olmOCR uses a vision-first pipeline that renders PDF pages to images, queries Gemini to detect column counts, and generates semantic HTML via Claude to preserve exact reading order before validating sequence with automated tests.**

The allenai/olmocr repository implements a robust, LLM-driven approach to **multi-column PDF layouts and reading order detection** that sidesteps traditional text-extraction heuristics. By treating layout recognition as a computer vision task rather than a parsing problem, the pipeline accurately identifies complex column structures and enforces top-to-bottom, left-to-right text flow. This article breaks down the implementation details, source file paths, and code patterns used to achieve reliable document structure preservation.

## Rendering PDF Pages to PNG Images

The pipeline begins by converting PDF pages into rasterized images that vision-enabled language models can analyze. In [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py), the `render_pdf_to_base64png` function handles this conversion.

The function rasterizes each page with a maximum dimension of **2048 pixels**, returning a base64-encoded PNG string suitable for API transmission. This image-first approach eliminates noise from traditional PDF text extraction and makes visual layout elements—columns, headers, sidebars—explicitly visible to the model.

```python
from olmocr.data.renderpdf import render_pdf_to_base64png

png_b64 = render_pdf_to_base64png(
    pdf_path="my_document.pdf",
    page_num=1,                 # 1-indexed page number

    target_longest_image_dim=2048,
)

```

## Layout Analysis with Gemini

Once rasterized, the image undergoes structured layout analysis via Google's Gemini-Flash model. The implementation resides in [`olmocr/bench/miners/mine_reading_order.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_reading_order.py) (lines 44-84), where the `analyze_document_layout` function constructs a specific prompt querying: *"How many columns are used in the main text document layout?"*

Gemini returns a JSON response containing a deterministic `num_columns` field alongside auxiliary flags for articles, inserts, and drop-caps. This structured schema eliminates ambiguous heuristics by forcing explicit numerical answers rather than interpretive descriptions.

```python
from olmocr.bench.miners.mine_reading_order import analyze_document_layout

layout, _ = analyze_document_layout(
    pdf_path="my_document.pdf",
    page_num=0,                 # 0-indexed for the function

    api_key="GEMINI_API_KEY",
)

print("Columns detected:", layout["num_columns"])

# Example output: Columns detected: 2

```

## HTML Generation with Column Preservation

When `num_columns` exceeds 1 or other multi-column indicators are present, the pipeline invokes Claude-Sonnet via [`olmocr/bench/miners/mine_multi_column.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_multi_column.py). The model receives explicit instructions to **"Preserve any multi-column layout using CSS flexbox or grid"**, generating clean semantic HTML that embeds the original visual structure.

The generated HTML uses responsive CSS (`display: flex; flex-direction: column;` or CSS Grid) to ensure downstream renderers faithfully reproduce the left-to-right, top-to-bottom reading flow. This guarantees that text extraction follows the natural visual order rather than the arbitrary storage order in the PDF file format.

```python
from olmocr.bench.miners.mine_multi_column import generate_html_from_image
from anthropic import Anthropic

client = Anthropic(api_key="ANTHROPIC_API_KEY")
html = generate_html_from_image(client, png_b64)

# The HTML will contain something like:

# <div class="multi-column" style="display:flex;">

#   <div class="column">…left‑column text…</div>

#   <div class="column">…right‑column text…</div>

# </div>

```

## Reading Order Test Generation

To validate that extracted text respects the intended sequence, the `generate_tests_from_html` function (also in [`mine_multi_column.py`](https://github.com/allenai/olmocr/blob/main/mine_multi_column.py)) creates concrete **before/after assertions**. This utility extracts sentences from the generated HTML, shuffles them, and produces order tests that explicitly assert correct sequence relationships.

These machine-readable tests encode the reading order into a verification suite, enabling automated validation that left-column content appears before right-column content in the final output. The CLI driver in [`scripts/qianfan_bench_convert.py`](https://github.com/allenai/olmocr/blob/main/scripts/qianfan_bench_convert.py) demonstrates this workflow and emphasizes the "top-to-bottom, left-to-right" reading-order rule.

```python
from olmocr.bench.miners.mine_multi_column import generate_tests_from_html

tests = generate_tests_from_html(html, pdf_id="my_document.pdf", page_num=1)
for t in tests:
    print(t["before"], "→", t["after"])

```

## Why This Vision-First Approach Works

**Image-based layout visibility** – Rendering to PNG makes column boundaries and visual hierarchy explicit to multimodal LLMs, avoiding the coordinate parsing errors common in traditional PDF extraction libraries.

**Deterministic column detection** – Gemini's JSON schema forces a concrete `num_columns` integer response, removing ambiguity from heuristic layout analysis.

**CSS-based structure preservation** – By embedding column counts in responsive HTML layouts rather than plain text, the system maintains spatial relationships through downstream rendering engines like Playwright or standard browsers.

**Automated order validation** – The generation of explicit before/after test pairs creates a machine-checkable contract that the extracted text follows natural reading order, even for complex layouts with insets or sidebars.

## Summary

- **olmOCR** processes multi-column PDFs through a four-stage pipeline: rasterization, Gemini-based layout analysis, Claude-driven HTML generation, and automated test creation.
- The `render_pdf_to_base64png` function in [`olmocr/data/renderpdf.py`](https://github.com/allenai/olmocr/blob/main/olmocr/data/renderpdf.py) converts pages to 2048px PNG images for vision model consumption.
- [`olmocr/bench/miners/mine_reading_order.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_reading_order.py) queries Gemini for explicit column counts via structured JSON responses.
- Multi-column documents are converted to semantic HTML using CSS Flexbox or Grid in [`olmocr/bench/miners/mine_multi_column.py`](https://github.com/allenai/olmocr/blob/main/olmocr/bench/miners/mine_multi_column.py) to preserve visual reading order.
- The `generate_tests_from_html` function creates before/after assertions that validate the correct text sequence programmatically.

## Frequently Asked Questions

### How does olmOCR determine the number of columns in a PDF page?

olmOCR sends a rasterized PNG of the page to Google's Gemini-Flash model with a structured prompt asking specifically for the column count. The model returns a JSON object containing a `num_columns` field, providing a deterministic integer value rather than an interpretive description. This approach avoids the edge cases and heuristics common in traditional PDF parsing libraries.

### Why does the pipeline use different LLMs for layout detection and HTML generation?

The pipeline uses Gemini-Flash for initial layout analysis because its JSON schema capabilities provide reliable structured output for column counting. For HTML generation, it employs Claude-Sonnet because of its superior performance in generating clean, semantic markup with precise CSS layout instructions. This division of labor optimizes for each model's strengths: Gemini for structured data extraction and Claude for complex markup generation.

### What ensures that the extracted text follows the correct reading order?

olmOCR enforces correct reading order through CSS-based HTML generation that explicitly structures content using Flexbox or Grid layouts, ensuring left columns render before right columns. Additionally, the `generate_tests_from_html` function creates automated tests that assert specific "before/after" relationships between text segments, providing machine-verifiable proof that the extraction preserves top-to-bottom, left-to-right flow.

### Can olmOCR handle complex layouts with sidebars, insets, or drop-caps?

Yes, the Gemini layout analysis detects auxiliary layout signals including articles, inserts, and drop-caps alongside the primary `num_columns` field. When these elements are present, the HTML generation instructions explicitly account for them, and the reading-order validation tests verify that the final text sequence respects the visual hierarchy regardless of how complex the original PDF layout appears.