How pdf-inspector Converts Extracted Text to Structured Markdown

pdf-inspector uses a four-stage pipeline—detection/extraction, layout analysis, classification/preprocessing, and a core conversion loop—to transform raw PDF text into clean, hierarchical Markdown, handling headings, lists, tables, and images according to geometric and typographic cues.

The open-source firecrawl/pdf-inspector library converts extracted PDF content into structured Markdown by analyzing geometric positions, font statistics, and layout signals rather than performing naive text extraction. Written in Rust, the tool preserves document hierarchy through a sophisticated pipeline that interprets visual structure to produce token-efficient output.

The Four-Stage Conversion Pipeline

The process to convert extracted text to structured Markdown follows a strict pipeline defined in src/lib.rs, where the public to_markdown function orchestrates four tightly-coupled stages.

Stage 1: PDF Type Detection and Text Extraction

The pipeline begins in src/detector.rs with detect_pdf_type, which determines whether the document is text-based, scanned, or mixed. For accessible PDFs, src/extractor/mod.rs executes extract_text_with_positions to return a flat list of TextItem structs. Each TextItem carries the Unicode string, font name, size, style flags, and physical coordinates (x, y, width, height).

The extractor then groups nearby TextItems into TextLine objects, preserving reading order while handling multi-column layouts. This geometric preservation ensures that spatial relationships survive into the final Markdown structure.

Stage 2: Layout and Font Analysis

In src/markdown/analysis.rs, the system computes font-size statistics via calculate_font_stats, building a histogram of font frequencies and calculating rarity scores (font_size_rarity). These statistics power the heading detection heuristics.

The analysis phase also identifies structural signals such as drop caps, list bullets, and table-of-contents (TOC) entries. Functions like bold_heading_level and detect_overused_struct_heading_levels suppress repeated structural headings, while is_toc_entry_line and is_toc_marker_heading filter out TOC lines to prevent false heading classification.

Stage 3: Classification and Pre-processing

The src/markdown/classify.rs module tags each TextLine with a semantic role—heading, list item, caption, code block, or prose—using heuristics like is_heading_fragment, is_list_item, and starts_with_bullet_marker.

Before conversion, src/markdown/preprocess.rs runs two critical cleanup operations:

• merge_drop_caps joins large initial letters with their following line to restore fragmented headings.
• merge_heading_lines collapses split heading fragments into single logical lines.

This pre-processing ensures that visual formatting quirks do not fragment the final Markdown structure.

Stage 4: Core Conversion and Post-Processing

The heart of the conversion lives in src/markdown/convert.rs. The system interleaves positioned blocks (tables and images) with text lines according to a logical chart-prose order defined by ChartProseOrder, using chart_stream_position to reconcile vertical ordering. This guarantees that tables appearing in newspaper-style columns maintain correct reading order.

During the conversion loop:

• Headings receive the appropriate number of # characters based on their detected level.
• List items are prefixed with - or * after format_list_item normalizes indentation.
• Code fragments and captions receive fenced blocks or italic styling.
• Tables are rendered by src/tables/format.rs, which builds Markdown pipe syntax, merges continuation rows, and caps column counts at 25.
• Images are emitted as ![](path) markdown links (extraction logic lives in extractor/xobjects.rs).

Finally, src/markdown/postprocess.rs executes clean_markdown to remove stray dot-leaders, fix hyphenation across line breaks, canonicalize URLs, and trim excess whitespace.

Key Heuristics for Structure Detection

pdf-inspector relies on typographic and spatial heuristics rather than PDF metadata tags to infer structure:

• Heading Level Detection: The system uses bold_heading_level to assign hierarchy based on font size rarity and bold styling, while detect_overused_struct_heading_levels suppresses repetitive header patterns that would otherwise clutter the output.
• TOC Filtering: TOC lines identified by is_toc_entry_line are stripped from heading consideration to avoid contaminating the document outline with table-of-contents entries.
• List Recognition: Bulleted lists are detected via starts_with_bullet_marker, which recognizes typographic bullets and normalized indentation levels.
• Geometric Ordering: The ChartProseOrder algorithm ensures that positioned elements like tables and images appear in the correct reading sequence relative to surrounding prose.

Using the pdf-inspector API

The public API exposed in src/lib.rs provides both high-level convenience functions and low-level control:

use pdf_inspector::{process_pdf, PdfOptions, ProcessMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Full pipeline: detection → extraction → markdown
    let result = process_pdf("sample.pdf")?;
    println!("Detected type: {:?}", result.pdf_type);
    if let Some(md) = result.markdown {
        println!("--- Markdown Output ---\n{md}");
    }

    // Custom options – only run the analyzer, no markdown generation
    let opts = PdfOptions::new()
        .mode(ProcessMode::Analyze)      // stop after analysis
        .pages([1, 2, 3]);               // process a subset of pages
    let analysis = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;
    println!("Page count: {}", analysis.page_count);
    Ok(())
}

For direct access to the conversion pipeline:

use pdf_inspector::markdown::{to_markdown_from_items, MarkdownOptions};

let items = pdf_inspector::extractor::extract_text("sample.pdf")?;
let md = to_markdown_from_items(&items, MarkdownOptions::default())?;
println!("{md}");

The internal to_markdown function (called by the high-level API) implements the four-stage flow:

pub fn to_markdown(items: &[TextItem]) -> Result<String> {
    // 1️⃣ extract → 2️⃣ analyze → 3️⃣ classify → 4️⃣ convert
    let lines = extractor::group_items_into_lines(items)?;
    let stats = markdown::analysis::calculate_font_stats(&lines);
    let classified = markdown::classify::classify_lines(&lines, &stats)?;
    let preprocessed = markdown::preprocess::run(classified)?;
    let raw_md = markdown::convert::run(preprocessed)?;
    Ok(markdown::postprocess::clean_markdown(raw_md))
}

Summary

• pdf-inspector converts extracted text to structured Markdown through four stages: detection/extraction (src/detector.rs, src/extractor/mod.rs), font/layout analysis (src/markdown/analysis.rs), classification/preprocessing (src/markdown/classify.rs, src/markdown/preprocess.rs), and conversion/post-processing (src/markdown/convert.rs, src/markdown/postprocess.rs).
• The system analyzes font-size rarity and geometric position rather than relying on PDF structural tags to infer headings, lists, and tables.
• Positioned blocks (tables, images) are interleaved with prose according to ChartProseOrder to preserve multi-column reading sequences.
• Post-processing in clean_markdown removes dot-leaders, fixes hyphenation, and canonicalizes URLs for clean output.
• The public API in src/lib.rs exposes both high-level convenience functions and low-level conversion primitives for custom workflows.

Frequently Asked Questions

How does pdf-inspector detect headings without PDF metadata?

pdf-inspector infers heading hierarchy through font statistics computed in src/markdown/analysis.rs. The calculate_font_stats function builds a histogram of font sizes, and the bold_heading_level heuristic assigns levels based on rarity scores (font_size_rarity) and bold styling. Overused structural patterns are suppressed by detect_overused_struct_heading_levels to avoid false positives, while TOC entries are explicitly filtered via is_toc_entry_line.

Can pdf-inspector handle complex layouts like multi-column documents?

Yes. The extraction engine in src/extractor/mod.rs groups TextItems into TextLine objects while preserving geometric coordinates. During conversion, src/markdown/convert.rs uses chart_stream_position and ChartProseOrder to interleave tables and images with prose lines according to their vertical position, ensuring that content flows correctly even in newspaper-style column layouts.

What table formatting limits does pdf-inspector enforce?

Table rendering occurs in src/tables/format.rs, which constructs standard Markdown pipe tables. The formatter automatically merges continuation rows to handle wrapped cell content and caps column counts at 25 columns to prevent excessively wide tables that would compromise readability in the final Markdown output.

How are drop caps and fragmented headings handled?

The src/markdown/preprocess.rs module contains specific logic to repair visual formatting artifacts. The merge_drop_caps function joins large initial letters with their subsequent lines, while merge_heading_lines collapses headings that were split across multiple TextItems during extraction. This pre-processing ensures that decorative typography does not fragment the logical Markdown structure.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}