PDF to Markdown Conversion Behavior in pdf-inspector: Default Pipeline Explained

The default pdf2md conversion in firecrawl/pdf-inspector executes a deterministic, fully-Rust pipeline that transforms PDFs into structured Markdown without OCR, completing in approximately 150 milliseconds for text-based documents by detecting document types, extracting positioned text lines, and applying layout-aware formatting rules.

The firecrawl/pdf-inspector repository provides a high-performance Rust library and CLI tool for converting PDF documents to Markdown. Understanding the default PDF to Markdown conversion behavior is essential for developers integrating this tool into AI preprocessing pipelines or document processing workflows. The default configuration prioritizes speed and structural accuracy over visual fidelity, bypassing OCR entirely for text-based documents while preserving logical document hierarchy through font analysis and geometric detection.

The Three-Stage Default Conversion Pipeline

The default conversion follows a deterministic three-stage architecture implemented across the src/detector.rs, src/extractor/, and src/markdown/ modules. This pipeline processes the PDF sequentially without redundant I/O operations.

Stage 1: Document Type Detection

Before extraction begins, src/detector.rs performs a rapid scan of the PDF's content streams to classify the document as TextBased, Scanned, ImageBased, or Mixed. This classification completes in approximately 10-50 milliseconds and produces a confidence score alongside a per-page list identifying which pages require OCR processing. The default conversion path skips OCR entirely when the document is classified as TextBased, enabling the sub-150ms processing times characteristic of the standard pipeline.

Stage 2: Content Extraction

The system loads the PDF once and walks every page's content stream via the modules in src/extractor/. Key components include:

• src/extractor/content_stream.rs: Parses raw PDF content streams
• src/extractor/fonts.rs: Resolves font dictionaries, CMap tables, and ToUnicode mappings
• src/extractor/layout.rs: Determines column structures and reading order

This stage gathers TextLine objects containing X/Y coordinates, font sizes, and style flags (bold, italic, underline). It simultaneously extracts table geometry through drawing operation analysis, identifies embedded images, and collects hyperlink annotations.

Stage 3: Markdown Generation

Finally, src/markdown/convert.rs transforms the extracted TextLine objects into Markdown syntax. The implementation applies deterministic rules to reconstruct document structure without preserving visual padding that would waste token space in downstream AI applications.

Default Markdown Formatting Rules

The src/markdown/convert.rs module applies specific heuristics to convert raw text positions into semantic Markdown elements:

• Headings: The system detects H1-H4 headings by analyzing font-size tiers relative to the most common body text size. Larger font sizes trigger heading classification based on relative thresholds.

• Lists: Bullet lists (recognizing •, -, *, and other markers) and numbered/lettered lists are identified through leading character analysis and indentation patterns.

• Code Blocks: Monospace font detection or explicit PDF StructRole::Code tags trigger fenced code block formatting in the output.

• Tables: The converter first attempts rectangle-based detection using union-find algorithms on drawing operations, falling back to heuristic text-alignment detection when geometric boundaries are ambiguous.

• Text Styling: Bold, italic, and underline formatting are applied when font names or ToUnicode mappings signal these styles in the source PDF.

• Hyperlinks: URL annotations are converted to standard Markdown link syntax [text](url).

• Page Breaks: The default configuration inserts HTML comments <!-- Page N --> at page boundaries to delineate pagination without disrupting Markdown rendering.

• Post-Processing: The pipeline collapses dot-leaders (table of contents lines), removes spurious hyphenation artifacts, strips page numbers from headers/footers, merges drop-cap characters with their subsequent text, and normalizes excessive whitespace.

Using the Default Conversion

The default behavior activates automatically when running pdf2md without optional flags. All code examples below demonstrate the standard, non-OCR conversion path.

Command Line Interface

Install the binary and convert documents with zero configuration:

# Install from crates.io or build from source

cargo install pdf-inspector

# Default conversion (no OCR, standard formatting)

pdf2md report.pdf

The command outputs Markdown with headings, lists, tables, and page-break comments as described above.

Rust Library API

Integrate the default pipeline programmatically using the process_pdf function from src/lib.rs:

use pdf_inspector::process_pdf;

fn main() -> Result<(), pdf_inspector::Error> {
    let result = process_pdf("report.pdf")?;
    
    println!("PDF type: {:?}", result.pdf_type);  // e.g., TextBased
    if let Some(md) = result.markdown {
        println!("{}", md);  // Default Markdown output
    }
    Ok(())
}

The process_pdf function automatically applies default PdfOptions and MarkdownOptions, triggering the three-stage pipeline without OCR.

Python and Node.js Bindings

Language bindings maintain identical default behavior:

import pdf_inspector

result = pdf_inspector.process_pdf("report.pdf")
print(result.pdf_type)      # "text_based", "scanned", etc.

print(result.markdown)      # Default Markdown string

import { readFileSync } from "fs";
import { processPdf } from "@firecrawl/pdf-inspector";

const pdf = readFileSync("report.pdf");
const result = processPdf(pdf);
console.log(result.pdfType);   // "TextBased", "Scanned", etc.
console.log(result.markdown);  // Default Markdown output

Key Source Files and Implementation Details

The default conversion behavior is distributed across specific modules:

• src/bin/pdf2md.rs: CLI entry point that parses arguments and invokes process_pdf with default MarkdownOptions

• src/lib.rs: Public API exposing process_pdf and PdfOptions builders that glue detection, extraction, and conversion stages together

• src/markdown/convert.rs: Core line-to-Markdown loop implementing heading, list, table, and code block detection heuristics

• src/detector.rs: Fast PDF-type classifier determining OCR necessity before extraction begins

• src/extractor/content_stream.rs, src/extractor/fonts.rs, src/extractor/layout.rs: Extract positioned TextLine objects, resolve font mappings, and determine reading order

Summary

• The default pipeline runs without OCR, completing in ~150ms for text-based PDFs by leveraging Rust's performance and single-pass extraction
• Three stages drive conversion: Document type detection (src/detector.rs), content extraction (src/extractor/), and Markdown generation (src/markdown/convert.rs)
• Structure is inferred from fonts and geometry: Headings derive from font-size tiers, lists from leading markers, tables from drawing operations or alignment heuristics
• Clean output is prioritized: The default settings collapse visual padding, remove headers/footers, and insert page-break comments rather than attempting pixel-perfect reproduction
• Multiple interfaces share behavior: The CLI pdf2md, Rust process_pdf, Python, and Node.js APIs all execute the same default pipeline when customization options are omitted

Frequently Asked Questions

Does the default conversion use OCR?

No. By default, pdf-inspector skips OCR entirely for documents classified as TextBased during the detection stage in src/detector.rs. The system only flags pages for OCR processing if the detector classifies them as Scanned, ImageBased, or Mixed, and even then, OCR requires explicit opt-in via command-line flags or API options. This design choice ensures sub-150ms conversion times for standard text-based PDFs.

How does pdf-inspector detect headings and lists?

The system analyzes TextLine objects extracted in src/extractor/layout.rs to identify document structure. Headings are detected through font-size tier analysis in src/markdown/convert.rs, comparing each line's font size against the most common body text size to assign H1-H4 levels. Lists are recognized by scanning for bullet markers (•, -, *) or numbered/lettered prefixes at the beginning of text lines, then grouping items by indentation levels.

What file handles the core Markdown transformation?

The primary logic resides in src/markdown/convert.rs. This module implements the line-to-Markdown conversion loop, applying default rules for headings, lists, code blocks, tables, and inline styling. It receives structured TextLine objects from the extraction stage and outputs clean Markdown, handling post-processing steps like hyphenation removal and whitespace normalization before returning the final string.

Can I customize the default Markdown output?

Yes, though the defaults are optimized for token efficiency. The process_pdf function in src/lib.rs accepts optional configuration structs that modify behavior. Command-line users can pass flags like --compact to reduce whitespace further or --pages to limit conversion to specific page ranges. However, omitting these flags executes the standard pipeline described above, which preserves structural elements while minimizing visual noise.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}