Why pdf-inspector Parses a PDF Only Once When Loading a Single Document

pdf-inspector parses PDF files only once per document load to share a single lopdf::Document instance across detection, extraction, and markdown generation stages, eliminating redundant I/O and guaranteeing consistent decoded state.

The architecture of firecrawl/pdf-inspector is intentionally built around single-pass PDF loading. This design choice—explicitly documented in the source code—drives performance gains and state consistency that would be impossible with multiple parse passes. Here's how it works and why it matters for your PDF processing pipelines.

How Single-Pass Loading Works

The core mechanism lives in src/lib.rs, where the public entry point process_pdf_with_options orchestrates the entire flow.

At line 90, the document is loaded into memory:

// From src/lib.rs#L90-L95
let bytes = std::fs::read(path)?;
let doc = lopdf::Document::load_mem(&bytes)?;

This doc variable—a lopdf::Document holding the complete PDF object graph, streams, and cross-reference tables—is then passed to process_document. The same instance routes to both the detector (for metadata analysis) and the extractor (for text and markdown generation). A code comment explicitly declares this intent: "The document is loaded once and shared between detection AND extraction."

Why This Design Delivers Performance

Eliminates Redundant Parsing Costs

Calling lopdf::Document::load_mem dominates runtime for large PDFs. By parsing once, pdf-inspector ensures that detect-pdf (metadata-only) and pdf2md (full extraction) share comparable start-up costs despite performing vastly different amounts of work. The CPU work of decoding object streams, resolving indirect references, and building the cross-reference table happens exactly once.

Enables Memory-Efficient Sharing

The Document object encapsulates:

• PDF object graph — all indirect objects and their generations
• Stream data — compressed content streams for pages and resources
• Cross-reference tables — mapping object numbers to file offsets
• Decoded font descriptors — /Font objects with embedded /ToUnicode CMaps

Keeping one instance in memory allows the detector to query page counts and resource dictionaries while the extractor streams text operators—without re-reading or re-decoding identical objects.

Guarantees Consistent Decoded State

PDF features like embedded /ToUnicode CMaps, font descriptors, and structure trees are expensive to resolve. Single-pass loading ensures every pipeline stage sees identical decoded font information. This prevents mismatches that could emerge if the file were reparsed independently—where subtle timing or caching differences might yield different character mappings.

Code Examples: Single-Parse Behavior in Practice

All public APIs preserve the parse-once semantics:

// Full detection + extraction (default mode)
let result = pdf_inspector::process_pdf("sample.pdf")?;

// Fast metadata-only detection (still parses once, but skips text extraction)
let meta = pdf_inspector::detect_pdf("sample.pdf")?;

// Custom options – document parsed once regardless of configuration
let opts = PdfOptions::new()
    .mode(ProcessMode::Full)
    .pages([1, 2, 3]);           // limit to specific pages
let custom = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;

// Per-page markdown extraction – single parse serves all pages
let pages = pdf_inspector::extract_pages_markdown("sample.pdf", None)?;
for page in pages.pages {
    println!("--- Page {} ---\n{}", page.page + 1, page.markdown);
}

Even when limiting output to specific pages or processing modes, the underlying document is never reloaded.

Key Files Enforcing Parse-Once Architecture

File	Role in single-pass design
src/lib.rs	Defines process_pdf_with_options; creates the sole lopdf::Document and routes it to downstream stages
src/detector.rs	Implements detect_from_document, accepting the shared Document for metadata queries
src/extractor/mod.rs	Orchestrates text extraction using the same Document instance supplied to the detector
src/extractor/content_stream.rs	Parses content streams against the already-loaded Document for font and resource lookups
src/tounicode.rs	Builds font-to-Unicode mappings from the shared Document; reused across the entire pipeline

These components enforce the "load-once, share-everywhere" contract that gives pdf-inspector its speed and reliability.

Summary

• Single lopdf::Document instance is created in src/lib.rs and shared across all processing stages
• Performance gain comes from eliminating redundant load_mem calls, which dominate runtime for large files
• Memory efficiency is achieved by holding one object graph, one set of decoded streams, and one cross-reference table
• State consistency guarantees that detectors and extractors resolve fonts, CMaps, and resources identically
• Explicit design choice is documented in source comments to prevent accidental double-parsing in future extensions

Frequently Asked Questions

Does pdf-inspector ever re-parse a PDF during processing?

No. Once process_pdf_with_options calls lopdf::Document::load_mem, the resulting Document is passed by reference through detection and extraction without reloading. Even page-limited extractions operate on the already-loaded structure.

Why not parse on demand for each operation?

Re-parsing would multiply I/O and CPU costs linearly with pipeline stages. More critically, independent parses could yield inconsistent font decoding if /ToUnicode CMaps or font descriptors resolve differently between passes. The single-parse design trades modest upfront memory for guaranteed consistency.

How does this affect processing of very large PDFs?

Memory usage scales with the document's object graph size, not with pipeline complexity. For extremely large files, lopdf provides memory-mapped alternatives, but pdf-inspector's single-parse approach already minimizes peak memory by avoiding duplicate structure copies.

Can I force re-parsing for fresh state?

The public API intentionally prevents this. If you need isolated processing of the same file, call the API separately—each invocation creates its own Document. This boundary ensures predictable resource management without hidden shared state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}