# How pdf-inspector Converts Extracted PDF Text to Markdown: A 4-Stage Pipeline Explained

> Discover how pdf-inspector converts extracted PDF text to Markdown through a 4-stage pipeline. Learn about document type detection, layout analysis, element classification, and token streaming for clean, hierarchical Markdown.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-08-31

---

**pdf-inspector transforms raw PDF text into structured Markdown by detecting document type, analyzing layout and typography, classifying structural elements, and streaming a conversion loop that emits clean, hierarchical Markdown tokens.**

pdf-inspector is a Rust-based PDF parsing engine developed by Firecrawl that converts PDF documents into Markdown format. This article explains how pdf-inspector converts extracted text to Markdown by walking through its four-stage pipeline, examining the actual source files from the `firecrawl/pdf-inspector` repository where this conversion logic is implemented.

## Stage 1: PDF Type Detection and Text Extraction

The conversion process begins in [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) and [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs). The `detect_pdf_type` function first determines whether the PDF contains a usable text layer, is scanned, or mixed. This detection step prevents wasted processing on image-only documents.

Once detected, `extract_text_with_positions` produces a flat list of `TextItem` structs. Each `TextItem` carries:

- Unicode string content
- Font name, size, and style flags
- Physical coordinates (`x`, `y`, `width`, `height`)

The extractor then groups nearby `TextItem`s into `TextLine` objects, preserving reading order while handling multi-column layouts. This positional metadata becomes critical for the layout analysis that follows.

## Stage 2: Layout and Font Analysis

In [`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs), pdf-inspector computes **font statistics** that drive structural inference. The `calculate_font_stats` function builds a histogram of font-size frequencies across the document. From this, it derives `font_size_rarity` scores—uncommon large fonts indicate potential headings.

The analysis module also implements:

- `bold_heading_level` – assigns heading levels based on bold-only formatting
- `detect_overused_struct_heading_levels` – suppresses repeated structural headings (e.g., page headers styled as `<H2>`)
- `is_toc_entry_line` and `is_toc_marker_heading` – identifies table-of-contents lines to avoid false heading detection

This statistical approach allows pdf-inspector to adapt to document-specific typographic conventions rather than relying on fixed thresholds.

## Stage 3: Classification and Pre-processing

The pipeline moves to [`src/markdown/classify.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/classify.rs) and [`src/markdown/preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/preprocess.rs) for semantic labeling and cleanup.

**Classification** ([`classify.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/classify.rs)) tags each `TextLine` with structural roles:

- `is_heading_fragment` – detects split heading lines
- `is_list_item` and `starts_with_bullet_marker` – identifies unordered lists
- Caption and code block detection via heuristic patterns

**Pre-processing** ([`preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/preprocess.rs)) then repairs common PDF artifacts:

- `merge_drop_caps` joins large initial letters with following lines
- `merge_heading_lines` collapses fragmented headings into single lines

This stage ensures that downstream conversion receives coherent, properly labeled text blocks.

## Stage 4: Core Conversion Loop and Post-Processing

The final transformation happens in [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) and [`src/markdown/postprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/postprocess.rs).

### Chart-Prose Order Interleaving

The [`convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/convert.rs) module handles **positioned blocks**—tables and images that must be interleaved with text at correct vertical positions. The `ChartProseOrder` system uses `chart_stream_position` to reconcile the logical reading order, ensuring that tables in newspaper-style columns appear where readers expect them.

### Markdown Token Emission

For each element in the sorted stream, the conversion loop emits appropriate Markdown syntax:

| Element | Markdown Output | Implementation |
|---------|---------------|----------------|
| Headings | `#` to `######` prefix | Level determined by font rarity and bold analysis |
| List items | `-` or `*` prefix | `format_list_item` normalizes indentation |
| Code blocks | Triple backtick fences | Detected via monospace font patterns |
| Tables | Pipe-delimited syntax | [`src/tables/format.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/format.rs) renders with 25-column cap, merges continuation rows |
| Images | `![](path)` syntax | Extraction handled in [`extractor/xobjects.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/xobjects.rs) |

### Final Cleanup

`clean_markdown` in [`postprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/postprocess.rs) performs terminal fixes:

- Removes dot-leaders from TOC lines
- Repairs hyphenation across line breaks
- Canonicalizes URL formatting
- Trims excess whitespace

## Public API and Usage

All stages are orchestrated from [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) through the `to_markdown` function:

```rust
pub fn to_markdown(items: &[TextItem]) -> Result<String> {
    // 1️⃣ extract → 2️⃣ analyze → 3️⃣ classify → 4️⃣ convert
    let lines = extractor::group_items_into_lines(items)?;
    let stats = markdown::analysis::calculate_font_stats(&lines);
    let classified = markdown::classify::classify_lines(&lines, &stats)?;
    let preprocessed = markdown::preprocess::run(classified)?;
    let raw_md = markdown::convert::run(preprocessed)?;
    Ok(markdown::postprocess::clean_markdown(raw_md))
}

```

### Full Pipeline Example

```rust
use pdf_inspector::{process_pdf, PdfOptions, ProcessMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Full pipeline: detection → extraction → markdown
    let result = process_pdf("sample.pdf")?;
    println!("Detected type: {:?}", result.pdf_type);
    if let Some(md) = result.markdown {
        println!("--- Markdown Output ---\n{md}");
    }

    // Custom options – only run the analyzer, no markdown generation
    let opts = PdfOptions::new()
        .mode(ProcessMode::Analyze)      // stop after analysis
        .pages([1, 2, 3]);               // process a subset of pages
    let analysis = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;
    println!("Page count: {}", analysis.page_count);
    Ok(())
}

```

### Low-Level Conversion API

```rust
use pdf_inspector::markdown::{to_markdown_from_items, MarkdownOptions};

let items = pdf_inspector::extractor::extract_text("sample.pdf")?;
let md = to_markdown_from_items(&items, MarkdownOptions::default())?;
println!("{md}");

```

## Summary

pdf-inspector converts PDF text to Markdown through a carefully sequenced pipeline:

- **Detection and extraction** ([`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs), [`extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/mod.rs)) identifies PDF type and extracts positioned text items
- **Layout analysis** ([`analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/analysis.rs)) computes font statistics and rarity scores to infer document structure
- **Classification and pre-processing** ([`classify.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/classify.rs), [`preprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/preprocess.rs)) labels lines semantically and repairs PDF artifacts like drop-caps
- **Conversion and post-processing** ([`convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/convert.rs), [`postprocess.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/postprocess.rs)) streams Markdown tokens in correct reading order and cleans final output

The result is **token-efficient Markdown** that preserves headings, lists, tables, images, and multi-column reading order without the noise typical of raw PDF text extraction.

## Frequently Asked Questions

### What PDF types can pdf-inspector handle?

pdf-inspector handles text-based PDFs, scanned documents, and mixed documents through its detection system in [`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs). For scanned PDFs without text layers, the engine would require OCR preprocessing—though the current implementation focuses on documents with extractable text content.

### How does pdf-inspector distinguish headings from regular text?

Headings are detected through **font-size rarity analysis** in [`analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/analysis.rs). Unusually large fonts receive higher rarity scores, which combine with bold formatting heuristics (`bold_heading_level`) to assign appropriate heading levels. The system also suppresses overused structural headings and excludes TOC lines from heading classification.

### Can pdf-inspector preserve complex table structures?

Yes. Tables are detected as positioned blocks and rendered by [`src/tables/format.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/format.rs) into Markdown pipe tables. The formatter handles continuation rows, caps column counts at 25 for readability, and maintains proper vertical ordering through the `ChartProseOrder` system that interleaves tables with surrounding prose.

### Is the conversion customizable?

Partially. The `PdfOptions` builder exposed in [`lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/lib.rs) allows selecting processing modes (full conversion versus analysis-only) and specifying page subsets. Lower-level APIs like `to_markdown_from_items` accept `MarkdownOptions` for controlling output behavior, though the core heuristics remain fixed to ensure consistent structural inference.