# How pdf-inspector Converts Extracted Text to Structured Markdown

> Discover how pdf-inspector converts extracted text to structured Markdown. Learn about its four-stage pipeline for headings, lists, tables, and images.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: how-to-guide
- Published: 2026-08-13

---

**pdf-inspector** uses a four-stage pipeline—detection/extraction, layout analysis, classification/preprocessing, and a core conversion loop—to transform raw PDF text into clean, hierarchical Markdown, handling headings, lists, tables, and images according to geometric and typographic cues.

The open-source `firecrawl/pdf-inspector` library converts extracted PDF content into structured Markdown by analyzing geometric positions, font statistics, and layout signals rather than performing naive text extraction. Written in Rust, the tool preserves document hierarchy through a sophisticated pipeline that interprets visual structure to produce token-efficient output.

## The Four-Stage Conversion Pipeline

The process to convert extracted text to structured Markdown follows a strict pipeline defined in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs), where the public `to_markdown` function orchestrates four tightly-coupled stages.

### Stage 1: PDF Type Detection and Text Extraction

The pipeline begins in [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) with `detect_pdf_type`, which determines whether the document is text-based, scanned, or mixed. For accessible PDFs, [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) executes `extract_text_with_positions` to return a flat list of `TextItem` structs. Each `TextItem` carries the Unicode string, font name, size, style flags, and physical coordinates (`x`, `y`, `width`, `height`).

The extractor then groups nearby `TextItem`s into `TextLine` objects, preserving reading order while handling multi-column layouts. This geometric preservation ensures that spatial relationships survive into the final Markdown structure.

### Stage 2: Layout and Font Analysis

In [`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs), the system computes **font-size statistics** via `calculate_font_stats`, building a histogram of font frequencies and calculating rarity scores (`font_size_rarity`). These statistics power the heading detection heuristics.

The analysis phase also identifies structural signals such as drop caps, list bullets, and table-of-contents (TOC) entries. Functions like `bold_heading_level` and `detect_overused_struct_heading_levels` suppress repeated structural headings, while `is_toc_entry_line` and `is_toc_marker_heading` filter out TOC lines to prevent false heading classification.

### Stage 3: Classification and Pre-processing

The [`src/markdown/classify.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/classify.rs) module tags each `TextLine` with a semantic role—heading, list item, caption, code block, or prose—using heuristics like `is_heading_fragment`, `is_list_item`, and `starts_with_bullet_marker`.

Before conversion, [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) runs two critical cleanup operations:

- `merge_drop_caps` joins large initial letters with their following line to restore fragmented headings.
- `merge_heading_lines` collapses split heading fragments into single logical lines.

This pre-processing ensures that visual formatting quirks do not fragment the final Markdown structure.

### Stage 4: Core Conversion and Post-Processing

The heart of the conversion lives in [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs). The system interleaves **positioned blocks** (tables and images) with text lines according to a logical **chart-prose order** defined by `ChartProseOrder`, using `chart_stream_position` to reconcile vertical ordering. This guarantees that tables appearing in newspaper-style columns maintain correct reading order.

During the conversion loop:

- Headings receive the appropriate number of `#` characters based on their detected level.
- List items are prefixed with `-` or `*` after `format_list_item` normalizes indentation.
- Code fragments and captions receive fenced blocks or italic styling.
- Tables are rendered by [`src/tables/format.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/format.rs), which builds Markdown pipe syntax, merges continuation rows, and caps column counts at 25.
- Images are emitted as `![](path)` markdown links (extraction logic lives in [`extractor/xobjects.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/xobjects.rs)).

Finally, [`src/markdown/postprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/postprocess.rs) executes `clean_markdown` to remove stray dot-leaders, fix hyphenation across line breaks, canonicalize URLs, and trim excess whitespace.

## Key Heuristics for Structure Detection

pdf-inspector relies on typographic and spatial heuristics rather than PDF metadata tags to infer structure:

- **Heading Level Detection**: The system uses `bold_heading_level` to assign hierarchy based on font size rarity and bold styling, while `detect_overused_struct_heading_levels` suppresses repetitive header patterns that would otherwise clutter the output.
- **TOC Filtering**: TOC lines identified by `is_toc_entry_line` are stripped from heading consideration to avoid contaminating the document outline with table-of-contents entries.
- **List Recognition**: Bulleted lists are detected via `starts_with_bullet_marker`, which recognizes typographic bullets and normalized indentation levels.
- **Geometric Ordering**: The `ChartProseOrder` algorithm ensures that positioned elements like tables and images appear in the correct reading sequence relative to surrounding prose.

## Using the pdf-inspector API

The public API exposed in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) provides both high-level convenience functions and low-level control:

```rust
use pdf_inspector::{process_pdf, PdfOptions, ProcessMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Full pipeline: detection → extraction → markdown
    let result = process_pdf("sample.pdf")?;
    println!("Detected type: {:?}", result.pdf_type);
    if let Some(md) = result.markdown {
        println!("--- Markdown Output ---\n{md}");
    }

    // Custom options – only run the analyzer, no markdown generation
    let opts = PdfOptions::new()
        .mode(ProcessMode::Analyze)      // stop after analysis
        .pages([1, 2, 3]);               // process a subset of pages
    let analysis = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;
    println!("Page count: {}", analysis.page_count);
    Ok(())
}

```

For direct access to the conversion pipeline:

```rust
use pdf_inspector::markdown::{to_markdown_from_items, MarkdownOptions};

let items = pdf_inspector::extractor::extract_text("sample.pdf")?;
let md = to_markdown_from_items(&items, MarkdownOptions::default())?;
println!("{md}");

```

The internal `to_markdown` function (called by the high-level API) implements the four-stage flow:

```rust
pub fn to_markdown(items: &[TextItem]) -> Result<String> {
    // 1️⃣ extract → 2️⃣ analyze → 3️⃣ classify → 4️⃣ convert
    let lines = extractor::group_items_into_lines(items)?;
    let stats = markdown::analysis::calculate_font_stats(&lines);
    let classified = markdown::classify::classify_lines(&lines, &stats)?;
    let preprocessed = markdown::preprocess::run(classified)?;
    let raw_md = markdown::convert::run(preprocessed)?;
    Ok(markdown::postprocess::clean_markdown(raw_md))
}

```

## Summary

- **pdf-inspector** converts extracted text to structured Markdown through four stages: detection/extraction ([`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs), [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs)), font/layout analysis ([`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs)), classification/preprocessing ([`src/markdown/classify.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/classify.rs), [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs)), and conversion/post-processing ([`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs), [`src/markdown/postprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/postprocess.rs)).
- The system analyzes **font-size rarity** and **geometric position** rather than relying on PDF structural tags to infer headings, lists, and tables.
- **Positioned blocks** (tables, images) are interleaved with prose according to `ChartProseOrder` to preserve multi-column reading sequences.
- Post-processing in `clean_markdown` removes dot-leaders, fixes hyphenation, and canonicalizes URLs for clean output.
- The public API in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) exposes both high-level convenience functions and low-level conversion primitives for custom workflows.

## Frequently Asked Questions

### How does pdf-inspector detect headings without PDF metadata?

pdf-inspector infers heading hierarchy through **font statistics** computed in [`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs). The `calculate_font_stats` function builds a histogram of font sizes, and the `bold_heading_level` heuristic assigns levels based on rarity scores (`font_size_rarity`) and bold styling. Overused structural patterns are suppressed by `detect_overused_struct_heading_levels` to avoid false positives, while TOC entries are explicitly filtered via `is_toc_entry_line`.

### Can pdf-inspector handle complex layouts like multi-column documents?

Yes. The extraction engine in [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) groups `TextItem`s into `TextLine` objects while preserving geometric coordinates. During conversion, [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) uses `chart_stream_position` and `ChartProseOrder` to interleave tables and images with prose lines according to their vertical position, ensuring that content flows correctly even in newspaper-style column layouts.

### What table formatting limits does pdf-inspector enforce?

Table rendering occurs in [`src/tables/format.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/format.rs), which constructs standard Markdown pipe tables. The formatter automatically merges continuation rows to handle wrapped cell content and **caps column counts at 25 columns** to prevent excessively wide tables that would compromise readability in the final Markdown output.

### How are drop caps and fragmented headings handled?

The [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) module contains specific logic to repair visual formatting artifacts. The `merge_drop_caps` function joins large initial letters with their subsequent lines, while `merge_heading_lines` collapses headings that were split across multiple `TextItem`s during extraction. This pre-processing ensures that decorative typography does not fragment the logical Markdown structure.