pdf-inspector Performance Benchmarks: Speed-First PDF Processing Explained

pdf-inspector processes PDFs at approximately 2.35 ms per document on modern hardware, achieving throughput of ~400 PDFs per second without OCR.

pdf-inspector is a Rust-based library designed for high-speed PDF to Markdown conversion. Unlike tools that rely on heavyweight ML models or OCR, it uses deterministic algorithms and smart short-circuiting to minimize latency and resource consumption. This guide breaks down the performance benchmarks, implementation details, and optimization strategies found in the firecrawl/pdf-inspector source code.

Core Performance Benchmarks

According to the official benchmark methodology in docs/benchmarking.md, pdf-inspector demonstrates consistent speed across repeated test runs:

Metric	Value	Hardware
Total processing time (200 PDFs)	0.470 s	Apple M4 Pro
Per-document latency	≈2.35 ms	Apple M4 Pro
Single-threaded throughput	~400 PDF/s	Modern laptop
Classification time	10–50 ms	Per document
Extraction + Markdown conversion	<3 ms	Text-based PDFs

These figures represent no-OCR processing of text-based PDFs. The median speed remains stable across five full-corpus benchmark runs, indicating predictable performance for production workloads.

Architecture Decisions That Drive Speed

Single-Pass Document Loading

The library eliminates redundant parsing through shared internal representation. When you call process_pdf() in src/lib.rs, the document is loaded once—whether from a file path via load_document_from_path or from memory via load_document_from_mem—and the same PdfDocument struct is reused by both the classifier and extractor.

// src/lib.rs - Public API handles single-load orchestration
pub fn process_pdf<P: AsRef<Path>>(path: P) -> Result<PdfResult, PdfError>
pub fn process_pdf_with_options<P: AsRef<Path>>(
    path: P,
    options: ProcessingOptions
) -> Result<PdfResult, PdfError>

This design choice removes the common pattern where PDF tools parse the same file multiple times for different operations.

Fast Classification via Stream Sampling

The src/detector.rs module implements sub-50 ms PDF type detection by sampling content streams rather than analyzing entire documents:

• Samples a small subset of streams to classify PDFs as Text, Scanned, Image-based, or Mixed
• Caches classification results so the extractor can skip OCR entirely when unnecessary
• Runs in 10–50 ms per document according to benchmarking data

This early exit is critical for throughput: OCR pipelines typically add 100–1000x latency, and pdf-inspector avoids them for the majority of documents.

Three-Tiered Table Detection

Table extraction in pdf-inspector uses short-circuit evaluation ordered by computational cost, as implemented across three source files:

1. Rectangle-based detection (src/tables/detect_rects.rs) — Uses drawing operation union-find; fastest path for most tables
2. Heuristic alignment detection (src/tables/detect_heuristic.rs) — Scans for aligned text columns when rectangles fail
3. Line-grid detection (src/tables/detect_lines.rs) — Full grid analysis only when tiers 1–2 fail

The majority of PDF tables match tier 1 or 2, avoiding the expensive line-following operations required by traditional PDF table extractors.

O(N) Column Detection Using Histogram Valleys

Multi-column newspaper layouts are handled in src/extractor/layout.rs through horizontal projection histograms:

1. Build histogram of text item x-coordinates per page
2. Identify valley regions (minimal text density)
3. Use valleys as column boundaries
4. Assign reading order within each column

This algorithm runs in linear time over text items—far cheaper than constraint-satisfaction or ML-based layout engines used by alternatives.

Memory Efficiency

The data structures in src/types.rs minimize allocation overhead:

// src/types.rs - Lightweight text representation
pub struct TextItem {
    pub content: String,
    pub x: f64,
    pub y: f64,
    pub width: f64,
    pub height: f64,
    pub font_index: u16,
    pub underlined: bool,
    // No large intermediate canvases, no glyph rasterization
}

pub struct PdfRect {
    pub x1: f64,
    pub y1: f64,
    pub x2: f64,
    pub y2: f64,
}

The library maintains zero large intermediate buffers—no pixel arrays, no full-page rasterization, and no ML tensor allocations.

Dependency Footprint

Cargo.toml declares a single external crate:

[dependencies]
lopdf = "0.32"

The lopdf crate provides low-level PDF parsing without pulling in:

• Heavyweight PDF rendering engines (no pdfium, poppler)
• ML frameworks (no torch, tensorflow, onnx)
• OCR engines (no tesseract bindings in core library)

This keeps binary size small and startup time minimal—critical for serverless and CLI usage patterns.

Measuring Performance Yourself

CLI Timing

# Measure single-PDF processing time

time pdf2md sample.pdf > /dev/null

Rust: Explicit Instrumentation

use std::time::Instant;
use pdf_inspector::process_pdf;

fn main() -> anyhow::Result<()> {
    let start = Instant::now();
    let result = process_pdf("sample.pdf")?;
    let elapsed = start.elapsed();
    
    println!("PDF type: {:?}", result.pdf_type);
    println!("Markdown length: {}", 
        result.markdown.as_ref().map(|s| s.len()).unwrap_or(0));
    println!("Processing time: {:.2?}", elapsed);
    
    Ok(())
}

Python: Batch Benchmarking

import time
import pdf_inspector

def process_batch(paths):
    start = time.time()
    for p in paths:
        r = pdf_inspector.process_pdf(p)
        # Access r.pdf_type, r.markdown as needed

    elapsed = time.time() - start
    print(f"Processed {len(paths)} PDFs in {elapsed:.2f}s")

# Benchmark run

process_batch(["doc1.pdf", "doc2.pdf", "doc3.pdf"])

Scalability Characteristics

Deployment Pattern	Expected Throughput	Key Factor
Single-threaded loop	~400 PDF/s	CPU-bound parsing
Multi-threaded (Rayon)	~800–1200 PDF/s	Lock-free document processing
Multi-process	1000s of PDF/s	Linear core scaling
Container/serverless	Baseline ~400 PDF/s	No external service dependencies

Because pdf-inspector is pure Rust with no external services, it scales linearly with available CPU cores. No GPU, no cloud API calls, no network latency.

Debugging Performance with Structured Logging

The library supports fine-grained profiling without recompilation via RUST_LOG:

# Debug only the layout engine

RUST_LOG=pdf_inspector::extractor::layout=debug pdf2md slow.pdf

# Debug full extraction pipeline

RUST_LOG=pdf_inspector=debug pdf2md slow.pdf

# Production: minimal logging

RUST_LOG=warn pdf2md batch/*.pdf

Configuration details are documented in docs/debugging.md.

Optimization Best Practices

1. Pre-classify to skip OCR — Call detection first; only run external OCR on PdfType::Scanned pages
2. Reuse PdfDocument — Use process_pdf once rather than separate classify/extract calls
3. Batch similar documents — Amortize startup costs across homogeneous workloads
4. Profile with targeted logging — Enable debug only for suspected bottleneck modules

Summary

• pdf-inspector achieves 2.35 ms per document on Apple M4 Pro hardware for text-based PDFs
• Single-pass loading in src/lib.rs eliminates redundant parsing
• Stream sampling detection in src/detector.rs enables sub-50 ms classification with OCR bypass
• Three-tiered table detection short-circuits expensive algorithms for typical documents
• O(N) histogram-based column detection in src/extractor/layout.rs replaces heavyweight layout engines
• Minimal dependency footprint (single lopdf crate) ensures fast startup and small binaries

Frequently Asked Questions

What hardware was used for the official pdf-inspector benchmarks?

The documented benchmarks in docs/benchmarking.md were conducted on an Apple M4 Pro. The 0.470 second result for 200 PDFs (2.35 ms/document) represents single-threaded processing without OCR. Performance scales linearly with CPU single-thread performance; expect proportionally faster results on newer hardware and slower results on constrained cloud instances.

Does pdf-inspector performance degrade with scanned PDFs?

Yes, but gracefully. The core library does not perform OCR—it classifies documents in src/detector.rs and returns PdfType::Scanned for image-based content. Your application can then route these to an external OCR engine. This design keeps pdf-inspector's latency bounded: classification remains 10–50 ms even for scanned documents, and you pay OCR costs only when actually needed.

How does pdf-inspector compare to Python-based PDF extractors?

pdf-inspector's Rust implementation and minimal dependencies provide 10–100x lower latency than typical Python PDF pipelines. Python tools often load heavy dependencies (PyMuPDF, pdfplumber, layoutparser) with 100+ ms import times alone. pdf-inspector's 2.35 ms per document includes full parsing, extraction, and Markdown generation—competitive with just the import overhead of many alternatives.

Can I improve throughput beyond the ~400 PDF/s benchmark?

Yes. The ~400 PDF/s figure represents a conservative single-threaded baseline. Through rayon parallelization or multi-process architectures, production deployments achieve thousands of PDFs per second. Because the library has no global state and no external service dependencies, it parallelizes perfectly across CPU cores without coordination overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}