How pdf-inspector Optimizes Document Loading to Share Data Between Detection and Extraction

pdf-inspector eliminates redundant PDF parsing by loading the document once into a shared lopdf::Document that both the detection and extraction phases consume without re-reading bytes or re-initializing parsers.

The firecrawl/pdf-inspector Rust library is designed for high-throughput PDF processing where every millisecond and memory allocation matters. Its core architectural decision—centralizing document loading around a single in-memory representation—ensures that classification logic and text extraction operate on identical parsed data. This article explains exactly how this optimization works, with references to the actual source implementation.

Single-Pass Loading in the Entry Point

The public API surface starts with process_pdf_with_options in [src/lib.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs). This function accepts raw PDF bytes and immediately constructs a lopdf::Document—the canonical parsed representation from the lopdf crate.

use pdf_inspector::{process_pdf_with_options, PdfOptions};

// Detect and extract in a single call – the PDF is parsed only once.
let result = process_pdf_with_options(
    "my_document.pdf",
    PdfOptions::new()               // default: auto-mode detection + extraction
).expect("failed to process");

// The `result` contains both detection metadata (pages_routed_to_ocr, etc.)
// and the final Markdown output.
println!("{}", result.markdown);

Once constructed, this Document instance travels through the entire pipeline. No subsequent code path re-opens the file, re-parses the cross-reference table, or reconstructs the object tree.

Detection Pipeline: Zero-Redundancy Classification

The detector.rs module receives the pre-loaded Document directly via process_pdf_with_options → detector::detect_pdf_type. According to the pdf-inspector source code, this function analyzes:

• Page objects – to count pages and assess content complexity
• Font dictionaries – to determine if embedded fonts contain extractable text
• Content streams – to distinguish actual text operators from vector graphics or bitmap masks

Because the detector works on the same lopdf::Document created at entry, it performs zero additional I/O. The classification into TextBased, Scanned, Mixed, or other types completes entirely in-memory using data structures already resident in RAM.

Extraction Pipeline: Reusing Parsed Objects

After detection finishes, control flows to extractor/mod.rs (also in [src/extractor/mod.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs)). This module receives the identical Document instance that the detector just analyzed.

The extractor walks the exact same object tree:

• Pages that the detector inspected for content type
• Font dictionaries that the detector evaluated for text presence
• Content streams that the detector probed for operator types

This reuse is critical for performance. The extractor can instantly access objects without re-parsing, and any intermediate calculations (page dimensions, font encodings) remain cached from the detection phase.

Lazy Heavy-Weight Resource Loading

The optimization extends beyond the lopdf::Document itself. pdf-inspector employs lazy initialization for expensive subsystems: PDFium (the Chromium PDF rendering engine used for OCR) and machine learning models are instantiated only when the detector flags pages as requiring OCR.

The test suite in [src/vision/pipeline.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs) contains auto_mode_does_not_load_pdfium_or_models_for_clean_pdf, which empirically verifies this behavior. A clean, text-based PDF never triggers PDFium initialization or model loading, keeping memory footprint minimal and startup latency near-zero.

Manual Two-Step Flow for Advanced Use Cases

When you need detection results before deciding on extraction strategy, pdf-inspector exposes lower-level functions that maintain the same sharing semantics:

// Manual two-step flow when you need the detection result first.
let bytes = std::fs::read("my_document.pdf").unwrap();

// 1️⃣ Detect PDF type (no extra loading)
let det = pdf_inspector::detect_pdf_type(&bytes).unwrap();

// 2️⃣ If OCR is needed, the extractor will lazily load PDFium.
let extraction = pdf_inspector::extract_pages_markdown_mem(&bytes, None).unwrap();
println!("{}", extraction.markdown);

Even in this decomposed pattern, the underlying implementation avoids redundant parsing. Both detect_pdf_type and extract_pages_markdown_mem internally construct a single Document for their respective operations, and the lazy loader ensures OCR infrastructure remains dormant for text-native documents.

Performance Impact of Shared Document Loading

By coupling detection and extraction around a shared lopdf::Document, pdf-inspector achieves three measurable benefits:

• Minimized I/O – File bytes are read exactly once, regardless of pipeline complexity
• Reduced CPU overhead – PDF structure parsing (cross-reference tables, object streams, compression decoding) executes exactly once per document
• Low memory footprint – A single parsed representation serves dual purposes instead of maintaining parallel copies

These characteristics make pdf-inspector suitable for high-throughput scenarios: server-side document processing, batch conversion pipelines, and real-time extraction services where per-document overhead directly impacts throughput.

Summary

• process_pdf_with_options in src/lib.rs creates one lopdf::Document that feeds both detection and extraction
• detector.rs classifies PDF type using the pre-parsed document with no additional loading
• extractor/mod.rs reuses the same in-memory object tree for Markdown generation
• Lazy initialization of PDFium and ML models ensures clean PDFs avoid heavy subsystems entirely
• Test coverage in src/vision/pipeline.rs validates that optimization paths execute as designed

Frequently Asked Questions

How does pdf-inspector avoid parsing PDFs twice?

pdf-inspector constructs a single lopdf::Document at the entry point in src/lib.rs. Both the detection logic in detector.rs and the extraction logic in extractor/mod.rs receive this same instance, so the PDF's cross-reference table, object tree, and content streams are parsed exactly once. No function in the pipeline re-opens or re-reads the source bytes.

What happens if detection determines OCR is unnecessary?

When detector::detect_pdf_type classifies a PDF as text-based, the extraction phase proceeds without ever initializing PDFium or loading machine learning models. The test auto_mode_does_not_load_pdfium_or_models_for_clean_pdf in src/vision/pipeline.rs confirms this lazy behavior, keeping memory and startup costs minimal for the majority case of digitally-born documents.

Can I use detection and extraction separately while maintaining optimization?

Yes. The lower-level API exposes detect_pdf_type and extract_pages_markdown_mem for manual orchestration. While each function constructs its own Document internally, neither performs redundant work, and the lazy loader ensures heavy OCR infrastructure activates only when detection results demand it.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}