# Detecting and Extracting Multi-Column Newspaper Layouts from PDFs with pdf-inspector

> Easily detect and extract multi column newspaper layouts from PDFs with pdf inspector. Our pipeline preserves natural reading order without machine learning.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: how-to-guide
- Published: 2026-08-10

---

**`pdf-inspector` automatically detects multi-column newspaper layouts using a two-stage histogram and XY-cut fallback pipeline that preserves natural reading order without machine learning.**

The `pdf-inspector` crate (maintained by Firecrawl) converts PDF text into structured Markdown while handling complex page geometries that break simpler extraction tools. For newspaper-style documents, the challenge lies in **column detection**—determining how text flows across vertical gutters before the markdown conversion stage can assemble the correct reading sequence.

## How Column Detection Works in pdf-inspector

The core algorithm lives in [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) within the `detect_columns` function. It processes each page through eight sequential stages:

### 1. Collect Page-Level Text Items

First, the pipeline filters `TextItem` objects using `crate::extractor::is_text_layout_item`. This removes image placeholders and artifacts that would distort the spatial histogram.

### 2. Build a Horizontal Projection Histogram

- Page width is divided into **2 pt bins** (`BIN_WIDTH`)
- Items wider than **60% of page width** are excluded as spanning elements (titles, full-width paragraphs)
- Remaining items increment bins across their horizontal span

### 3. Identify Empty Valleys (Gutters)

- Noise threshold = **15% of maximum bin count** (`NOISE_FRACTION = 0.15`)
- Consecutive low-count bins become candidate valleys
- Valleys must satisfy **minimum 8 pt width** (`MIN_GUTTER_WIDTH`) and stay **5% away from page margins**

### 4. Relative-Valley Fallback for Justified Text

When gutters contain partial text from justified columns, absolute valleys disappear. The pipeline:

- Applies **5-bin moving average** smoothing
- Finds local minima **≥60% below surrounding peaks** (`CONTRAST_THRESHOLD`, `find_relative_valleys`)
- Keeps the deepest relative valley as fallback

### 5. Validate Valleys with Vertical Consistency (`validate_and_build_columns`)

Items are assigned to valley sides using **center-based** or **edge-based** logic (`center_assign` flag). Center-based assignment handles justified text where characters overhang gutters.

Validation rules:
- Both sides need **minimum items** (`MIN_ITEMS_PER_COLUMN`)
- Vertical overlap must cover **≥30% of page height** (`MIN_VERTICAL_SPAN_RATIO`)
- Sidebars allowed with **≥3 items**; dominant column needs full minimum
- Columns of mostly bullet-style list markers are rejected (`is_list_marker_column`)

### 6. Prose-Density Check (`columns_have_prose`)

Valid splits must resemble flowing text:
- Items grouped into lines by **y-proximity ≈ 3 pt**
- **≥40% of lines must span ≥45% of column width**
- Average items-per-line must not exceed **3.5** (table/form detection)

### 7. XY-Cut Fallback (`try_xy_cut_split`)

If histogram valleys fail, a simplified single-level XY-cut executes:

- Finds largest horizontal gap between right-edges and left-edges
- Gap must be **≥15 pt** and **≥10% from margins**
- Requires **10+ items** (major column), **3+ items** (minor column)
- Vertical overlap **≥20% of page height**

### 8. ColumnRegion Generation

Validated valleys become `ColumnRegion` structs. The extractor groups items per column, detects spanning lines (cross-column titles), and feeds ordered `TextLine`s to [`markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/markdown/convert.rs).

## Using pdf-inspector for Newspaper PDFs

### CLI Usage

```bash

# Basic Markdown extraction

pdf2md newspaper_issue.pdf

# Structured JSON for downstream processing

pdf2md newspaper_issue.pdf --json > issue.json

# Debug logging to trace column detection decisions

RUST_LOG=pdf_inspector::extractor::layout=debug pdf2md newspaper_issue.pdf

```

### Programmatic Rust API

```rust
use pdf_inspector::process_pdf_with_options;
use pdf_inspector::options::PdfProcessingOptions;

let opts = PdfProcessingOptions {
    table_detection: true,      // preserve tables, treat columns as prose
    xy_cut_fallback: false,     // disable fallback for clean PDFs
    ..Default::default()
};

let result = process_pdf_with_options("archive_issue.pdf", opts)?;
println!("{}", result.markdown);

```

## Key Source Files and Their Roles

| File | Responsibility |
|------|---------------|
| [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) | Histogram construction, valley validation, `validate_and_build_columns`, `try_xy_cut_split` |
| [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) | Consumes `ColumnRegion`s, assembles `TextLine` stream for Markdown |
| [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) | PDF classification (TextBased/Scanned/Mixed) that triggers newspaper-layout logic |
| [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) | Post-processing: drop-cap merging, header line reconciliation |
| [`src/bin/pdf2md.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/bin/pdf2md.rs) | CLI entry point wiring extractor → markdown modules |
| [`src/types.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/types.rs) | `TextItem`, `TextLine`, `effective_width` geometry helpers |

## Design Trade-offs

The pipeline prioritizes **speed over complexity**: single-pass histogram analysis with optional XY-cut avoids machine learning inference. This handles asymmetric layouts, sidebars, and justified multi-column text that naïve detectors—relying solely on fixed thresholds or simple whitespace gaps—typically misorder.

Constants like `BIN_WIDTH = 2.0`, `CONTRAST_THRESHOLD = 0.60`, and `MIN_VERTICAL_SPAN_RATIO = 0.30` are tuned for news print but can be adjusted at compilation for other document families.

## Summary

- **Histogram-based detection** with 2 pt bins and 60% width exclusion for spanning elements
- **Dual validation**: absolute valleys (15% noise floor) and relative valleys (60% contrast) for justified text
- **Prose-density filtering** prevents table/form misclassification via line-fill and items-per-line metrics
- **XY-cut fallback** recovers when gutters are partially obscured
- **Automatic execution** via `pdf2md` CLI or `process_pdf_with_options` Rust API

## Frequently Asked Questions

### Does pdf-inspector require machine learning for column detection?

No. The `detect_columns` function in [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) uses deterministic geometric analysis—horizontal projection histograms and XY-cut partitioning—without neural networks. This keeps extraction fast and deterministic.

### What happens when newspaper columns have uneven widths?

The `validate_and_build_columns` function accepts asymmetric layouts. The dominant column must satisfy `MIN_ITEMS_PER_COLUMN` while narrow sidebars need only 3 items. Vertical overlap checks (`MIN_VERTICAL_SPAN_RATIO = 0.30`) ensure both columns represent genuine parallel content streams.

### How does pdf-inspector distinguish multi-column text from tables?

The `columns_have_prose` test rejects column splits where:
- Average items-per-line exceeds 3.5 (table signature)
- Fewer than 40% of lines fill ≥45% of column width (fragmented content)

This filtering prevents tabular data from being misextracted as newspaper columns.

### Can I disable the XY-cut fallback if my PDFs are consistently clean?

Yes. Set `xy_cut_fallback: false` in `PdfProcessingOptions` when calling `process_pdf_with_options`. This skips `try_xy_cut_split` entirely, reducing runtime for documents where histogram valleys always succeed.