How pdf-inspector Converts Extracted PDF Text to Markdown: A 4-Stage Pipeline Explained

pdf-inspector transforms raw PDF text into structured Markdown by detecting document type, analyzing layout and typography, classifying structural elements, and streaming a conversion loop that emits clean, hierarchical Markdown tokens.

pdf-inspector is a Rust-based PDF parsing engine developed by Firecrawl that converts PDF documents into Markdown format. This article explains how pdf-inspector converts extracted text to Markdown by walking through its four-stage pipeline, examining the actual source files from the firecrawl/pdf-inspector repository where this conversion logic is implemented.

Stage 1: PDF Type Detection and Text Extraction

The conversion process begins in src/detector.rs and src/extractor/mod.rs. The detect_pdf_type function first determines whether the PDF contains a usable text layer, is scanned, or mixed. This detection step prevents wasted processing on image-only documents.

Once detected, extract_text_with_positions produces a flat list of TextItem structs. Each TextItem carries:

• Unicode string content
• Font name, size, and style flags
• Physical coordinates (x, y, width, height)

The extractor then groups nearby TextItems into TextLine objects, preserving reading order while handling multi-column layouts. This positional metadata becomes critical for the layout analysis that follows.

Stage 2: Layout and Font Analysis

In src/markdown/analysis.rs, pdf-inspector computes font statistics that drive structural inference. The calculate_font_stats function builds a histogram of font-size frequencies across the document. From this, it derives font_size_rarity scores—uncommon large fonts indicate potential headings.

The analysis module also implements:

• bold_heading_level – assigns heading levels based on bold-only formatting
• detect_overused_struct_heading_levels – suppresses repeated structural headings (e.g., page headers styled as <H2>)
• is_toc_entry_line and is_toc_marker_heading – identifies table-of-contents lines to avoid false heading detection

This statistical approach allows pdf-inspector to adapt to document-specific typographic conventions rather than relying on fixed thresholds.

Stage 3: Classification and Pre-processing

The pipeline moves to src/markdown/classify.rs and src/markdown/preprocess.rs for semantic labeling and cleanup.

Classification (classify.rs) tags each TextLine with structural roles:

• is_heading_fragment – detects split heading lines
• is_list_item and starts_with_bullet_marker – identifies unordered lists
• Caption and code block detection via heuristic patterns

Pre-processing (preprocess.rs) then repairs common PDF artifacts:

• merge_drop_caps joins large initial letters with following lines
• merge_heading_lines collapses fragmented headings into single lines

This stage ensures that downstream conversion receives coherent, properly labeled text blocks.

Stage 4: Core Conversion Loop and Post-Processing

The final transformation happens in src/markdown/convert.rs and src/markdown/postprocess.rs.

Chart-Prose Order Interleaving

The convert.rs module handles positioned blocks—tables and images that must be interleaved with text at correct vertical positions. The ChartProseOrder system uses chart_stream_position to reconcile the logical reading order, ensuring that tables in newspaper-style columns appear where readers expect them.

Markdown Token Emission

For each element in the sorted stream, the conversion loop emits appropriate Markdown syntax:

Element	Markdown Output	Implementation
Headings	# to ###### prefix	Level determined by font rarity and bold analysis
List items	- or * prefix	format_list_item normalizes indentation
Code blocks	Triple backtick fences	Detected via monospace font patterns
Tables	Pipe-delimited syntax	src/tables/format.rs renders with 25-column cap, merges continuation rows
Images	 syntax	Extraction handled in extractor/xobjects.rs

Final Cleanup

clean_markdown in postprocess.rs performs terminal fixes:

• Removes dot-leaders from TOC lines
• Repairs hyphenation across line breaks
• Canonicalizes URL formatting
• Trims excess whitespace

Public API and Usage

All stages are orchestrated from src/lib.rs through the to_markdown function:

pub fn to_markdown(items: &[TextItem]) -> Result<String> {
    // 1️⃣ extract → 2️⃣ analyze → 3️⃣ classify → 4️⃣ convert
    let lines = extractor::group_items_into_lines(items)?;
    let stats = markdown::analysis::calculate_font_stats(&lines);
    let classified = markdown::classify::classify_lines(&lines, &stats)?;
    let preprocessed = markdown::preprocess::run(classified)?;
    let raw_md = markdown::convert::run(preprocessed)?;
    Ok(markdown::postprocess::clean_markdown(raw_md))
}

Full Pipeline Example

use pdf_inspector::{process_pdf, PdfOptions, ProcessMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Full pipeline: detection → extraction → markdown
    let result = process_pdf("sample.pdf")?;
    println!("Detected type: {:?}", result.pdf_type);
    if let Some(md) = result.markdown {
        println!("--- Markdown Output ---\n{md}");
    }

    // Custom options – only run the analyzer, no markdown generation
    let opts = PdfOptions::new()
        .mode(ProcessMode::Analyze)      // stop after analysis
        .pages([1, 2, 3]);               // process a subset of pages
    let analysis = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;
    println!("Page count: {}", analysis.page_count);
    Ok(())
}

Low-Level Conversion API

use pdf_inspector::markdown::{to_markdown_from_items, MarkdownOptions};

let items = pdf_inspector::extractor::extract_text("sample.pdf")?;
let md = to_markdown_from_items(&items, MarkdownOptions::default())?;
println!("{md}");

Summary

pdf-inspector converts PDF text to Markdown through a carefully sequenced pipeline:

• Detection and extraction (detector.rs, extractor/mod.rs) identifies PDF type and extracts positioned text items
• Layout analysis (analysis.rs) computes font statistics and rarity scores to infer document structure
• Classification and pre-processing (classify.rs, preprocess.rs) labels lines semantically and repairs PDF artifacts like drop-caps
• Conversion and post-processing (convert.rs, postprocess.rs) streams Markdown tokens in correct reading order and cleans final output

The result is token-efficient Markdown that preserves headings, lists, tables, images, and multi-column reading order without the noise typical of raw PDF text extraction.

Frequently Asked Questions

What PDF types can pdf-inspector handle?

pdf-inspector handles text-based PDFs, scanned documents, and mixed documents through its detection system in detector.rs. For scanned PDFs without text layers, the engine would require OCR preprocessing—though the current implementation focuses on documents with extractable text content.

How does pdf-inspector distinguish headings from regular text?

Headings are detected through font-size rarity analysis in analysis.rs. Unusually large fonts receive higher rarity scores, which combine with bold formatting heuristics (bold_heading_level) to assign appropriate heading levels. The system also suppresses overused structural headings and excludes TOC lines from heading classification.

Can pdf-inspector preserve complex table structures?

Yes. Tables are detected as positioned blocks and rendered by src/tables/format.rs into Markdown pipe tables. The formatter handles continuation rows, caps column counts at 25 for readability, and maintains proper vertical ordering through the ChartProseOrder system that interleaves tables with surrounding prose.

Is the conversion customizable?

Partially. The PdfOptions builder exposed in lib.rs allows selecting processing modes (full conversion versus analysis-only) and specifying page subsets. Lower-level APIs like to_markdown_from_items accept MarkdownOptions for controlling output behavior, though the core heuristics remain fixed to ensure consistent structural inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}