How the pdf-inspector Layout Module Differentiates Newspaper vs. Tabular Reading Orders

The pdf-inspector layout engine distinguishes newspaper reading order from tabular order by detecting columns, grouping lines within each column, and applying density, balance, and prose heuristics to classify the layout type.

The pdf-inspector library from Firecrawl uses a sophisticated three-stage pipeline to determine whether a multi-column PDF page should be read as newspaper-style prose (columns processed sequentially) or as a table (rows interleaved across columns). This decision directly impacts how extracted text flows and how downstream table detection operates.

Column Detection: The Foundation of Layout Analysis

The process begins with detect_columns in src/extractor/layout.rs (lines 21–86). This function builds a horizontal occupancy histogram across the page, identifying gutter valleys that separate columns. It validates these valleys using absolute or relative thresholds and returns a vector of ColumnRegion objects describing each detected column.

// Detect columns on a page
let columns = detect_columns(&items, page_number, page_has_table);

Without reliable column detection, no downstream reading order classification is possible. The algorithm handles both clean gutter-based layouts and noisier scanned documents.

Per-Column Line Grouping and Prose Analysis

For each detected column, the engine groups items into rough text lines using Y-proximity clustering. The columns_have_prose helper (lines 557–573) performs critical measurements:

• Line count per column
• Full-line ratio: what portion of lines span ≥45% of column width

This data feeds directly into the newspaper-or-table decision.

The is_newspaper_layout Decision Function

The core classification logic resides in is_newspaper_layout, declared at lines 1832–1834 of src/extractor/layout.rs. This function applies four interlocking heuristics:

1. Density Floor (15-Line Minimum)

Both columns must contain at least 15 lines (commented at lines 1903–1904). Sparse columns fail immediately and default to tabular treatment. This prevents fragmentary sidebars or captions from triggering false newspaper detection.

2. Balanced Line Counts

Columns with similar line counts (within a factor of 2) indicate independent text flows characteristic of newspaper layouts. Severely imbalanced counts suggest asymmetric tables or marginalia.

3. Prose Evidence via columns_have_prose

Each column undergoes deeper scrutiny in columns_have_prose (lines 669–674, 595–604, 668–670). A column qualifies as prose when it satisfies:

• Minimum width: ≥120 points
• Minimum lines: ≥8 lines
• Full-line ratio: ≥40% of lines cover ≥45% of column width, OR a consecutive run of ≥6 full lines exists
• Item density: average items per line ≤3.5 (prose has fewer, larger items than table cells)

4. Unanimous Column Agreement

All detected columns must satisfy the prose criteria for newspaper classification. If any column fails, the engine falls back to tabular reading order where rows are interleaved across columns.

// Decide reading order based on column analysis
let is_newspaper = is_newspaper_layout(&per_column_lines, &columns);
let reading_order = if is_newspaper { "newspaper" } else { "tabular" };

Test Suite Validation

The implementation is validated by targeted unit tests in src/extractor/mod.rs:

• test_newspaper_layout_detection (lines 3000–3045): Creates two dense columns with matching Y positions, expects true
• test_newspaper_layout_misaligned_baselines (lines in test block): Confirms detection works even with independent column baselines
• Sparse column test (lines 3094–3135): Verifies the 15-line floor correctly returns false

These tests ensure the heuristics perform robustly across real-world document variations.

Key Source Files

File	Responsibility
src/extractor/layout.rs	Column detection, valley validation, is_newspaper_layout, columns_have_prose heuristics
src/extractor/mod.rs	Public API exposure and comprehensive unit tests
src/tables/mod.rs	Consumes reading-order decisions for table processing logic

Summary

• Column detection via detect_columns establishes the geometric structure
• Line grouping within columns measures density and fill patterns
• Density floor: 15+ lines required per column
• Balance check: line counts within 2× factor
• Prose verification: width, full-line ratio, and item density thresholds
• Unanimous requirement: all columns must qualify as prose for newspaper classification
• Falls back to tabular reading order when any heuristic fails

Frequently Asked Questions

How does pdf-inspector handle documents with more than two columns?

The is_newspaper_layout function evaluates all detected columns uniformly. Every column must pass the density, balance, and prose criteria. Multi-column newspaper layouts with three or more columns follow the same unanimous voting logic—any column that fails prose detection triggers tabular fallback.

What happens when columns have significantly different line counts?

Severely imbalanced line counts (beyond a 2× ratio) indicate the columns are not independent text flows. This commonly occurs in tables with header rows spanning one column, or documents with main text plus narrow sidebar comments. The engine treats such cases as tabular layouts.

Why does the prose detection use a 3.5 items-per-line threshold?

Prose paragraphs typically contain continuous text runs with few discrete items per typographic line. Table cells, conversely, often contain single words, numbers, or short phrases—yielding higher item counts. The ≤3.5 threshold distinguishes these patterns empirically, as implemented in columns_have_prose at lines 668–670.

Can the newspaper detection thresholds be configured?

The current implementation in src/extractor/layout.rs uses hardcoded constants: 15-line floor, 120pt minimum width, 45% full-line threshold, 40% full-line ratio, 6-line consecutive run, and 3.5 items-per-line maximum. These values reflect tuned heuristics for general document collections. Users seeking different behavior must modify the source constants directly.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}