How pdf-inspector Analyzes Column Layouts in PDF Documents: Histogram-Based Detection in Rust

pdf-inspector uses horizontal projection histograms and geometric analysis of text baselines to detect column boundaries, classify layouts as newspaper or tabular, and reconstruct reading order in multi-column PDFs.

The firecrawl/pdf-inspector repository provides a Rust-based extraction engine that converts PDF documents into structured Markdown. A critical challenge in this pipeline is accurately identifying column layouts—whether a page contains a single text flow, newspaper-style columns, or tabular data. The analysis hinges on geometric interpretation of TextItem objects derived from the raw PDF content stream.

Horizontal Projection Histograms and Gap Detection

The foundation of column detection resides in src/extractor/layout.rs, specifically within the detect_columns function. This implementation projects all text baselines onto the X-axis to build a spatial occupancy map.

The algorithm aggregates TextItem coordinates into histogram buckets of approximately 2 points each. Peaks in this histogram represent densely populated vertical regions (columns), while valleys indicate potential gutter spaces. A valley qualifies as a column boundary only when the gap exceeds the configurable threshold COLUMN_GAP_MIN (set to approximately 40 points by default) and demonstrates sufficient contrast with surrounding peaks. This geometric approach allows the system to distinguish between intentional column separations and minor spacing variations within a single text block.

ColumnRegion Struct and Item Assignment

Once gaps are identified, the system constructs column boundaries using the ColumnRegion struct defined in the layout module. Each region stores explicit geometric bounds:

ColumnRegion {
    x_min: f64,  // Left boundary in PDF points
    x_max: f64,  // Right boundary in PDF points
}

The list of ColumnRegion objects is sorted left-to-right to establish reading order. The helper function assign_to_best_overlap then maps every TextItem to the column with which it shares the greatest horizontal overlap. Items that span multiple columns—such as centered titles or page headers—are identified through pre-masking logic that removes "spanning lines" from the histogram calculation. This prevents wide, short elements from corrupting the column boundary detection.

Newspaper vs. Tabular Layout Classification

After establishing column boundaries, is_newspaper_layout analyzes the distribution of content across columns to classify the page structure. The function examines per_column_lines data to determine if the layout follows asymmetric newspaper conventions or balanced tabular arrangements.

A page is classified as newspaper layout when more than 60% of items cluster in a single column or when the column-wise reading order exhibits high asymmetry (such as a main text column with side annotations). Otherwise, the system treats the page as tabular, feeding the detected column boundaries into src/tables/mod.rs via try_build_table_from_columns to reconstruct borderless tables. This classification determines whether the extraction engine preserves strict columnar reading order or interprets the content as structured data rows.

Integration in the Extraction Pipeline

The column detection workflow integrates at the library level through process_pdf_with_options in src/lib.rs. This entry point processes each page sequentially, invoking extractor::detect_columns after building the raw text stream.

The resulting column metadata is stored in PageInfo.pages_with_columns (defined in src/types.rs), making the geometric analysis available to downstream modules. These modules use the column data for reading-order reconstruction, table detection, and final Markdown conversion, ensuring that multi-column academic papers, financial reports, and magazine layouts render correctly as linear text.

Practical Implementation Examples

You can interact with the column detection system programmatically or via the command-line interface:

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::extractor::detect_columns;

fn analyze_pdf_columns(pdf_path: &str) -> Result<(), Box<dyn std::error::Error>> {
    let pdf_bytes = std::fs::read(pdf_path)?;
    let mut opts = pdf_inspector::ProcessOptions::default();
    let result = process_pdf_with_options(&pdf_bytes, &mut opts)?;
    
    for (page_no, page) in result.pages.iter().enumerate() {
        // Access raw TextItem baselines via layout_items
        let columns = detect_columns(&page.layout_items, page_no as u32, false);
        println!(
            "Page {} contains {} columns: {:?}",
            page_no + 1,
            columns.len(),
            columns
        );
    }
    Ok(())
}

For CLI users, the pdf2md binary exposes this functionality internally:

# Convert multi-column PDF to Markdown with automatic layout detection

pdf2md research-paper.pdf -o output.md

# Inspect detected column boundaries in JSON format

pdf2md financial-report.pdf --json | jq '.pages[].columns'

Summary

• Geometric histogram analysis in src/extractor/layout.rs projects text baselines onto the X-axis using ~2pt buckets to identify column gaps exceeding 40 points.
• ColumnRegion structs store precise boundary coordinates and are sorted left-to-right to establish proper reading order before item assignment.
• Pre-masking logic removes spanning elements (titles, headers) from histogram calculations to prevent boundary corruption.
• Layout classification distinguishes asymmetric newspaper layouts (>60% content in one column) from balanced tabular structures that trigger table detection pipelines.
• Pipeline integration occurs through process_pdf_with_options in src/lib.rs, with results stored in PageInfo.pages_with_columns for downstream Markdown conversion.

Frequently Asked Questions

How does pdf-inspector handle single-column documents?

Single-column documents naturally produce histograms without significant valleys exceeding the COLUMN_GAP_MIN threshold. The detect_columns function returns a single ColumnRegion spanning the full page width, and is_newspaper_layout typically classifies these as non-tabular unless specific asymmetric patterns are detected. The system treats this as a standard text flow without attempting multi-column reconstruction.

What is the default minimum gap threshold for column detection?

The default COLUMN_GAP_MIN threshold is approximately 40 points (roughly 0.55 inches or 14mm). This value filters out minor spacing variations within paragraphs while capturing intentional column gutters in standard document layouts. Users can adjust this parameter in the ProcessOptions configuration when processing documents with unusually narrow or wide column separations.

Can pdf-inspector detect tables using column information?

Yes. When is_newspaper_layout determines that columns are balanced rather than asymmetric, the system routes the detected ColumnRegion boundaries to try_build_table_from_columns in src/tables/mod.rs. This function attempts to reconstruct borderless tables by aligning text items into rows based on their vertical positions within the established column structure, effectively converting spatial columns into logical table cells.

How are multi-column headings handled during extraction?

Headings that span multiple columns are identified through pre-masking before histogram generation. The algorithm detects lines that cross detected gap boundaries while maintaining minimal height (indicating a single line of text rather than a paragraph). These spanning items are temporarily excluded from column assignment logic, then re-inserted at the appropriate position in the reading order based on their vertical coordinates, ensuring titles appear correctly above their respective column content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}