# How pdf-inspector Optimizes Document Loading to Share Data Between Detection and Extraction

> Discover how pdf-inspector optimizes document loading by using a single lopdf Document for detection and extraction, eliminating redundant parsing and improving efficiency.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: performance
- Published: 2026-09-02

---

**`pdf-inspector` eliminates redundant PDF parsing by loading the document once into a shared `lopdf::Document` that both the detection and extraction phases consume without re-reading bytes or re-initializing parsers.**

The `firecrawl/pdf-inspector` Rust library is designed for high-throughput PDF processing where every millisecond and memory allocation matters. Its core architectural decision—centralizing document loading around a single in-memory representation—ensures that classification logic and text extraction operate on identical parsed data. This article explains exactly how this optimization works, with references to the actual source implementation.

## Single-Pass Loading in the Entry Point

The public API surface starts with **`process_pdf_with_options`** in [[`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs). This function accepts raw PDF bytes and immediately constructs a `lopdf::Document`—the canonical parsed representation from the `lopdf` crate.

```rust
use pdf_inspector::{process_pdf_with_options, PdfOptions};

// Detect and extract in a single call – the PDF is parsed only once.
let result = process_pdf_with_options(
    "my_document.pdf",
    PdfOptions::new()               // default: auto-mode detection + extraction
).expect("failed to process");

// The `result` contains both detection metadata (pages_routed_to_ocr, etc.)
// and the final Markdown output.
println!("{}", result.markdown);

```

Once constructed, this `Document` instance travels through the entire pipeline. No subsequent code path re-opens the file, re-parses the cross-reference table, or reconstructs the object tree.

## Detection Pipeline: Zero-Redundancy Classification

The **[`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs)** module receives the pre-loaded `Document` directly via `process_pdf_with_options → detector::detect_pdf_type`. According to the `pdf-inspector` source code, this function analyzes:

- **Page objects** – to count pages and assess content complexity
- **Font dictionaries** – to determine if embedded fonts contain extractable text
- **Content streams** – to distinguish actual text operators from vector graphics or bitmap masks

Because the detector works on the same `lopdf::Document` created at entry, it performs **zero additional I/O**. The classification into *TextBased*, *Scanned*, *Mixed*, or other types completes entirely in-memory using data structures already resident in RAM.

## Extraction Pipeline: Reusing Parsed Objects

After detection finishes, control flows to **[`extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/mod.rs)** (also in [[`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs)](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs)). This module receives the identical `Document` instance that the detector just analyzed.

The extractor walks the exact same object tree:

- Pages that the detector inspected for content type
- Font dictionaries that the detector evaluated for text presence
- Content streams that the detector probed for operator types

This reuse is critical for performance. The extractor can instantly access objects without re-parsing, and any intermediate calculations (page dimensions, font encodings) remain cached from the detection phase.

## Lazy Heavy-Weight Resource Loading

The optimization extends beyond the `lopdf::Document` itself. `pdf-inspector` employs **lazy initialization** for expensive subsystems: **PDFium** (the Chromium PDF rendering engine used for OCR) and machine learning models are instantiated **only when the detector flags pages as requiring OCR**.

The test suite in [[`src/vision/pipeline.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs)](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs) contains `auto_mode_does_not_load_pdfium_or_models_for_clean_pdf`, which empirically verifies this behavior. A clean, text-based PDF never triggers PDFium initialization or model loading, keeping memory footprint minimal and startup latency near-zero.

## Manual Two-Step Flow for Advanced Use Cases

When you need detection results before deciding on extraction strategy, `pdf-inspector` exposes lower-level functions that maintain the same sharing semantics:

```rust
// Manual two-step flow when you need the detection result first.
let bytes = std::fs::read("my_document.pdf").unwrap();

// 1️⃣ Detect PDF type (no extra loading)
let det = pdf_inspector::detect_pdf_type(&bytes).unwrap();

// 2️⃣ If OCR is needed, the extractor will lazily load PDFium.
let extraction = pdf_inspector::extract_pages_markdown_mem(&bytes, None).unwrap();
println!("{}", extraction.markdown);

```

Even in this decomposed pattern, the underlying implementation avoids redundant parsing. Both `detect_pdf_type` and `extract_pages_markdown_mem` internally construct a single `Document` for their respective operations, and the lazy loader ensures OCR infrastructure remains dormant for text-native documents.

## Performance Impact of Shared Document Loading

By coupling detection and extraction around a shared `lopdf::Document`, `pdf-inspector` achieves three measurable benefits:

- **Minimized I/O** – File bytes are read exactly once, regardless of pipeline complexity
- **Reduced CPU overhead** – PDF structure parsing (cross-reference tables, object streams, compression decoding) executes exactly once per document
- **Low memory footprint** – A single parsed representation serves dual purposes instead of maintaining parallel copies

These characteristics make `pdf-inspector` suitable for high-throughput scenarios: server-side document processing, batch conversion pipelines, and real-time extraction services where per-document overhead directly impacts throughput.

## Summary

- **`process_pdf_with_options`** in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) creates one `lopdf::Document` that feeds both detection and extraction
- **[`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs)** classifies PDF type using the pre-parsed document with no additional loading
- **[`extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/mod.rs)** reuses the same in-memory object tree for Markdown generation
- **Lazy initialization** of PDFium and ML models ensures clean PDFs avoid heavy subsystems entirely
- **Test coverage** in [`src/vision/pipeline.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs) validates that optimization paths execute as designed

## Frequently Asked Questions

### How does pdf-inspector avoid parsing PDFs twice?

`pdf-inspector` constructs a single `lopdf::Document` at the entry point in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs). Both the detection logic in [`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs) and the extraction logic in [`extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/mod.rs) receive this same instance, so the PDF's cross-reference table, object tree, and content streams are parsed exactly once. No function in the pipeline re-opens or re-reads the source bytes.

### What happens if detection determines OCR is unnecessary?

When `detector::detect_pdf_type` classifies a PDF as text-based, the extraction phase proceeds without ever initializing PDFium or loading machine learning models. The test `auto_mode_does_not_load_pdfium_or_models_for_clean_pdf` in [`src/vision/pipeline.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs) confirms this lazy behavior, keeping memory and startup costs minimal for the majority case of digitally-born documents.

### Can I use detection and extraction separately while maintaining optimization?

Yes. The lower-level API exposes `detect_pdf_type` and `extract_pages_markdown_mem` for manual orchestration. While each function constructs its own `Document` internally, neither performs redundant work, and the lazy loader ensures heavy OCR infrastructure activates only when detection results demand it.