How PDF-Inspector Avoids Redundant I/O Between Detection and Extraction Stages

PDF-Inspector eliminates duplicate file reads by loading a PDF into a single in-memory lopdf::Document that is shared between both the detection and extraction pipelines.

The firecrawl/pdf-inspector Rust library processes PDFs through two distinct stages—first detecting document properties (type, page count, OCR requirements), then extracting structured text. Rather than parsing the file twice, the library constructs one Document instance and passes it to both stages. This design choice significantly reduces disk I/O and CPU overhead, especially for large PDFs or high-throughput applications.

The Single-Load Architecture

At the core of this optimization sits the process_pdf_with_options function in src/lib.rs. This public API entry point orchestrates the entire pipeline with a single file read.

/// Process a PDF file with custom options.
///
/// The document is loaded **once** and shared between detection and extraction.
pub fn process_pdf_with_options<P: AsRef<Path>>(
    path: P,
    options: PdfOptions,
) -> Result<PdfProcessResult, PdfError> {
    let start = ProcessingTimer::start();
    validate_pdf_file(&path)?;

    // Load the document once — shared by detection **AND** extraction.
    let (doc, page_count) =
        load_document_from_path_with_password(&path, options.password.as_deref())?;

    process_document(doc, page_count, options, start)
}

The (doc, page_count) tuple returned here becomes the foundation for all subsequent operations. Notice that doc is an owned lopdf::Document, not a path or buffer that would require re-parsing.

How the Document Is Loaded

The loading logic resides in load_document_from_path_with_password, which performs exactly one filesystem read:

/// Load a PDF file, decrypting with `password` if the file is encrypted.
pub(crate) fn load_document_from_path_with_password<P: AsRef<Path>>(
    path: P,
    password: Option<&str>,
) -> Result<(Document, u32), PdfError> {
    let buffer = std::fs::read(&path)?;
    load_document_from_mem_with_password(&buffer, password)
}

This function reads the entire file into a Vec<u8> buffer, then delegates to load_document_from_mem_with_password to handle decryption and PDF structure parsing. The resulting Document contains the fully parsed PDF cross-reference table, object tree, and page hierarchy—expensive data that would be wasteful to reconstruct.

Detection Stage: Zero Additional I/O

After loading, process_document immediately hands the doc reference to the detection pipeline:

let detection = detector::detect_from_document(&doc, page_count, &options.detection)?;

The detect_from_document function in src/detector.rs inspects the already-loaded structure to determine:

• PDF version and compliance level
• Presence of scanned pages requiring OCR
• Font embedding and encoding schemes
• Security settings and encryption status

Because the Document is passed by reference (&doc), the detection stage performs no file operations whatsoever. It traverses the in-memory object graph that lopdf has already built.

Extraction Stage: Reusing the Same Document

Following detection, extraction functions in src/extractor/mod.rs and its submodules receive the same &doc reference. For example, extractor::extract_positioned_text_from_doc operates directly on this shared document to:

• Decode content streams
• Map character codes to Unicode
• Reconstruct reading order and layout

Since the PDF's cross-reference table and stream objects remain accessible in memory, extraction proceeds without re-reading the source file or re-parsing the PDF's structural metadata.

Memory-Buffer Variants for Network Workflows

The same single-load pattern extends to in-memory APIs for scenarios where PDFs arrive from HTTP requests, databases, or message queues:

let bytes = std::fs::read("sample.pdf").unwrap();
let result = pdf_inspector::process_pdf_mem(&bytes).unwrap(); // one parse

Under the hood, process_pdf_mem calls load_document_from_mem_with_password, which bypasses std::fs::read but otherwise follows the identical path: parse once, share everywhere. Both detect_pdf_mem and the full process_pdf_mem pipeline use this same entry point.

Practical Usage Patterns

Detection-Only Mode (Fast Metadata)

When you need only document properties without text extraction, use the detect-only pathway to minimize work:

use pdf_inspector::{detect_pdf, PdfOptions, ProcessMode};

let result = detect_pdf("sample.pdf").unwrap(); // loads once, runs detection
println!("type: {:?}, pages: {}", result.pdf_type, result.page_count);

Even in this minimal mode, the document is fully parsed—there is no "lazy" or "partial" loading. However, the extraction stage is skipped entirely, saving the CPU cost of text decoding and layout analysis.

Full Detect-Then-Extract Pipeline

For complete processing, the default process_pdf function handles both stages with guaranteed single I/O:

use pdf_inspector::{process_pdf, PdfOptions};

let result = process_pdf("sample.pdf").unwrap(); // one load, shared doc
println!("{}", result.markdown.unwrap());

The PdfProcessResult contains both the detection metadata and the final markdown string, all derived from one lopdf::Document instance.

Key Implementation Files

File	Responsibility
src/lib.rs	Public API entry points, single-load orchestration in process_pdf_with_options
src/detector.rs	detect_from_document—consumes shared &Document for metadata extraction
src/extractor/mod.rs	Text extraction utilities that accept the same &Document reference
src/extractor/content_stream.rs	Low-level content stream parsing using the loaded document's object cache
src/extractor/layout.rs	Layout analysis operating on items extracted from the shared document

Summary

• One fs::read per file: The load_document_from_path_with_password function in src/lib.rs performs the sole filesystem read.
• Shared lopdf::Document: Both detector::detect_from_document and extractor functions receive the same in-memory document reference.
• No re-parsing: The PDF's cross-reference table, object tree, and page hierarchy are built exactly once.
• Symmetric APIs: File-based and memory-buffer variants (process_pdf/process_pdf_mem) follow identical single-load semantics.
• Performance benefit: Eliminates redundant I/O and CPU-intensive PDF parsing, especially critical for batch processing or large documents.

Frequently Asked Questions

How does PDF-Inspector handle password-protected PDFs without reading the file twice?

The password is passed through load_document_from_path_with_password and applied during the initial parse by load_document_from_mem_with_password. The decrypted Document is then shared with both detection and extraction stages. No second decryption or file read occurs.

Can I use detection results to decide whether to run extraction?

Yes. The process_pdf_with_options function accepts a PdfOptions struct that includes ProcessMode. You can inspect detection results from a preliminary call and conditionally invoke full extraction, or use PdfOptions to configure extraction behavior based on detected PDF type—all without reloading the document if you structure your own wrapper to cache the Document.

What happens if the PDF is corrupted or malformed?

load_document_from_path_with_password returns a PdfError during the initial parse if lopdf cannot interpret the file structure. Both detection and extraction are aborted since they depend on the successfully loaded Document. This fail-fast behavior prevents wasted work on unreadable inputs.

Is the same optimization available for streaming or chunked PDF sources?

The current implementation requires the complete file in memory (via std::fs::read or provided buffer) because lopdf needs random access to the PDF's cross-reference table. Streaming sources must be fully buffered before parsing; the single-load optimization then applies to the resulting in-memory document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}