How to Extract Text from PDF to Markdown Using Rust with pdf-inspector

The pdf-inspector crate provides a pure Rust pipeline that converts PDF documents into clean, token-efficient Markdown through detection, extraction, and structured conversion phases.

The firecrawl/pdf-inspector repository offers a comprehensive Rust library and CLI tool designed specifically for transforming PDF content into AI-friendly Markdown format. Built on top of the lopdf parser, this solution handles everything from text-based documents to scanned images without requiring external dependencies like Python or Node.js.

The pdf-inspector Pipeline Architecture

The library implements a sophisticated seven-stage pipeline that processes PDFs deterministically:

1. PDF Detection – The detector (src/detector.rs) analyzes the document once to classify it as text-based, scanned, image-based, or mixed. It identifies pages requiring OCR, detects encoding issues, and gathers layout metadata including tables and multi-column sections.

2. Document Loading – Documents open once using load_document_from_path_with_password or load_document_from_mem, with the handle shared between detection and extraction phases to eliminate redundant parsing.

3. Text Extraction – The extractor (src/extractor/mod.rs and sub-modules) traverses PDF content streams, resolves font mappings (including TrueType fallbacks), and generates TextItem structs containing raw text, positional data, styling, and MCID metadata for tagged PDFs.

4. Quality Verification – The text-quality module (src/text_quality.rs) applies heuristics to detect garbled text, CID-encoding failures, and decoding errors, flagging problematic pages for OCR processing.

5. Layout Analysis – The tables package (src/tables/*) executes three detection strategies (rectangle-based, line-based, heuristic) to identify tabular data, while the layout module (src/extractor/layout.rs) constructs column histograms to detect newspaper-style multi-column layouts.

6. Markdown Generation – The markdown crate (src/markdown/convert.rs) transforms positioned TextItems into Markdown using document-wide font statistics to infer heading hierarchies, merging drop-caps and applying compact formatting profiles.

7. Output Formatting – Results return as structured data containing the Markdown string, PDF classification, page metadata, and optional JSON representations.

Installation and Setup

Add the crate to your Cargo.toml:

[dependencies]
pdf-inspector = "0.1"

For command-line usage, install the binary:

cargo install pdf-inspector

Basic Usage Examples

Full Extraction with Default Options

The simplest approach processes the entire PDF using defaults. In src/lib.rs, the process_pdf function handles the complete pipeline:

use pdf_inspector::{process_pdf, PdfOptions, ProcessMode};

fn main() -> Result<(), pdf_inspector::PdfError> {
    // Use defaults (full extraction)
    let result = process_pdf("example.pdf")?;
    println!("PDF type: {:?}", result.pdf_type);
    if let Some(md) = result.markdown {
        println!("--- Markdown output ---\n{}", md);
    }
    Ok(())
}

This invokes ProcessMode::Full, running detection, extraction, and Markdown generation sequentially.

Detection-Only Mode

For scenarios requiring classification without full text extraction, use ProcessMode::DetectOnly:

use pdf_inspector::{detect_pdf_with_options, PdfOptions, ProcessMode};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = PdfOptions::new()
        .mode(ProcessMode::DetectOnly)
        .pages([1, 3, 5]);
    let info = detect_pdf_with_options("example.pdf", opts)?;
    println!("Detected type: {:?}, pages: {}", info.pdf_type, info.page_count);
    Ok(())
}

As implemented in src/detector.rs, this mode analyzes the PDF structure to determine if the document contains extractable text, scanned images, or hybrid content.

Per-Page Markdown Extraction

For hybrid OCR pipelines or page-specific processing, extract_pages_markdown returns individual Markdown strings per page:

use pdf_inspector::extract_pages_markdown;

fn main() -> Result<(), pdf_inspector::PdfError> {
    let result = extract_pages_markdown("report.pdf", None)?; // all pages
    for page_md in result.pages {
        println!("--- Page {} ---\n{}", page_md.page + 1, page_md.markdown);
    }
    Ok(())
}

This function maintains page boundaries, allowing selective reprocessing or alternative OCR handling for specific pages.

Command-Line Interface (CLI)

The pdf2md binary (src/bin/pdf2md.rs) exposes the library functionality through a convenient command-line interface:

# Full markdown extraction

pdf2md my_doc.pdf

# JSON output (metadata + markdown)

pdf2md my_doc.pdf --json

# Items-JSON (positioned TextItem list)

pdf2md my_doc.pdf --items-json > items.json

# Process only selected pages

pdf2md my_doc.pdf --select-pages 1,3,5-7

The CLI parses flags, constructs PdfOptions, and invokes process_pdf_with_options, formatting output as raw Markdown, JSON wrappers, or detailed item lists depending on the flags provided.

Key Components and Source Files

Understanding the source structure enables advanced customization:

• src/lib.rs – Defines the public API surface including PdfOptions, ProcessMode enum, and entry points process_pdf, detect_pdf, and process_pdf_with_options.

• src/bin/pdf2md.rs – CLI implementation handling argument parsing via standard Rust env args, instantiating options, and managing output serialization.

• src/detector.rs – Contains classification logic for PDF types and OCR requirements, calculating layout complexity scores.

• src/extractor/mod.rs – Core extraction engine parsing content streams, resolving font dictionaries, and generating TextItem collections with geometric data (PdfRect, PdfLine).

• src/markdown/convert.rs – Transforms extracted text items into Markdown, analyzing font statistics across the document to determine heading levels and structural hierarchy.

• src/text_quality.rs – Post-extraction validation detecting encoding errors and CID mapping issues.

• src/tables/ – Multi-strategy table detection algorithms identifying tabular structures through geometric analysis.

Advanced Configuration with PdfOptions

Fine-tune the extraction pipeline through the builder-pattern API defined in src/lib.rs:

use pdf_inspector::{process_pdf_with_options, PdfOptions, ProcessMode, MarkdownProfile};

let opts = PdfOptions::new()
    .mode(ProcessMode::Full)
    .pages(1..=10)  // Limit to first 10 pages
    .password("secret")
    .markdown_profile(MarkdownProfile::Compact);  // Reduce token count

let result = process_pdf_with_options("encrypted.pdf", opts)?;

ProcessMode variants include:

• Full – Complete detection, extraction, and Markdown generation
• DetectOnly – Classification without text extraction
• Analyze – Detailed structural analysis

The MarkdownProfile setting controls formatting density, optimizing output for either human readability or AI token efficiency.

Summary

• Pure Rust implementation eliminates external runtime dependencies, operating entirely on the lopdf parser foundation.
• Single-pass document loading through load_document_from_path_with_password ensures efficient memory usage by sharing file handles between detection and extraction phases.
• Multi-stage quality pipeline identifies encoding issues in src/text_quality.rs and routes problematic pages toward OCR workflows.
• Structural analysis detects tables via geometric algorithms and multi-column layouts through histogram analysis in src/extractor/layout.rs.
• Flexible output supports both programmatic access through process_pdf and CLI usage via pdf2md, with optional JSON serialization and per-page granularity.

Frequently Asked Questions

How does pdf-inspector handle scanned PDFs?

The detector (src/detector.rs) analyzes each page's content streams to determine if it contains extractable text or raster images. Pages identified as scanned or image-based are flagged with OCR recommendations, while the extraction pipeline (src/extractor/mod.rs) processes only text-based content. For mixed documents, the per-page extraction functions allow you to handle text pages in Rust while routing image pages to external OCR engines.

Can I extract text from password-protected PDFs?

Yes. The PdfOptions builder provides a .password() method that passes credentials to load_document_from_path_with_password. Both the library API and CLI support encrypted documents—the CLI accepts passwords via interactive prompts or environment variables, while programmatic usage allows direct string passing through the options struct.

What Markdown formatting options are available?

The src/markdown/convert.rs module generates Markdown using document-wide font statistics to infer heading hierarchies (H1-H6). The MarkdownProfile enum controls formatting density: standard mode preserves structural cues like spacing, while compact mode minimizes tokens for LLM consumption. The converter also handles edge cases like drop-caps, multi-column text reconstruction, and optional page-break markers.

How accurate is the table detection?

The tables package (src/tables/) implements a three-tier detection strategy: rectangle-based detection identifies explicit table borders, line-based analysis finds grid structures through vector graphics, and heuristic methods infer tabular data from text alignment patterns. This multi-strategy approach handles both explicit PDF table structures and visually formatted text tables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}