Processing PDF Files from a Memory Buffer with pdf-inspector

You can process PDFs directly from memory buffers in pdf-inspector using the detect_pdf_type_mem and extract_text_with_positions_mem functions, which both call load_document_from_mem to parse bytes without any filesystem I/O.

The pdf-inspector crate from Firecrawl is architected around a single-load pipeline: a PDF is parsed once into a lopdf::Document, and that same in-memory representation powers both type detection and text extraction. This design makes processing PDF files from a memory buffer straightforward and efficient, eliminating the need for temporary files when working with network streams, generated documents, or cached data.

Core Memory Buffer API

The public API exposes two function families for in-memory processing in src/lib.rs:

Family	Function	Input	Output
Detection	detect_pdf_type_mem	&[u8] buffer	PdfTypeResult (type, confidence, OCR hints)
Extraction	extract_text_with_positions_mem	&[u8] buffer	Vec<TextItem> with position, font, and text data

Both routes converge on load_document_from_mem, the internal parser entry point:

pub(crate) fn load_document_from_mem(buffer: &[u8])
    -> Result<(Document, u32), PdfError>
{
    load_document_from_mem_with_password(buffer, None)
}

This function handles malformed structure-tree names, repairs damaged containers, decrypts password-protected PDFs if needed, and returns the parsed Document plus page count. The implementation resides at src/lib.rs lines 84–93.

Pipeline Flow for Memory Buffers

PDF bytes (Vec<u8>)
   │
   ▼
load_document_from_mem → (Document, page_count)
   │
   ├──► detector::detect_pdf_type_mem      // fast classification
   │
   └──► extractor::extract_text_with_positions_mem
          ├──► font analysis (widths, CMap decoding)
          ├──► content stream parsing (operator state machine)
          ├──► XObject handling (Form XObjects)
          └──► layout engine (column detection, reading order)

All downstream modules—tables, markdown, and others—operate on the same Document instance. This guarantees zero redundant parsing and consistent state across detection and extraction phases.

Implementation Examples

Rust

use pdf_inspector::{
    detect_pdf_type_mem,
    extract_text_with_positions_mem,
    PdfOptions,
    ProcessMode,
};

fn process_from_memory(pdf_bytes: &[u8]) {
    // Fast type detection
    let detection = detect_pdf_type_mem(pdf_bytes)
        .expect("PDF type detection failed");
    println!("Detected: {:?} (confidence: {})", 
             detection.pdf_type, detection.confidence);

    // Full extraction with options
    let opts = PdfOptions::new()
        .mode(ProcessMode::Full);
    let extracted = extract_text_with_positions_mem(pdf_bytes, opts)
        .expect("Text extraction failed");

    // Convert to Markdown
    let markdown = pdf_inspector::to_markdown(&extracted).unwrap();
    println!("{}", markdown);
}

Python (PyO3 Bindings)

import pdf_inspector

# pdf_bytes: a bytes object from HTTP response, database, etc.

pdf_bytes = b"..."

# Detect PDF type without filesystem access

info = pdf_inspector.detect_pdf_type_mem(pdf_bytes)
print(f"{info.pdf_type}: {info.confidence}")

# Extract text with full pipeline

result = pdf_inspector.extract_text_with_positions_mem(pdf_bytes)
markdown = pdf_inspector.to_markdown(result)
print(markdown)

Node.js (N-API Bindings)

import { detectPdfTypeMem, extractTextWithPositionsMem, toMarkdown } from '@firecrawl/pdf-inspector';

// Buffer is Node's Uint8Array implementation
const pdfBytes = Buffer.from(response.data); // or readFileSync, fetch arrayBuffer, etc.

const detection = detectPdfTypeMem(pdfBytes);
console.log(detection.pdfType, detection.confidence);

const extracted = extractTextWithPositionsMem(pdfBytes);
const markdown = toMarkdown(extracted);
console.log(markdown);

WebAssembly (Browser)

import init, { processPdf } from '@firecrawl/pdf-inspector-wasm';

await init();  // Load WASM module

const response = await fetch('/document.pdf');
const pdfBytes = new Uint8Array(await response.arrayBuffer());

// High-level helper processes buffer directly
const result = processPdf(pdfBytes);
console.log(result.pdfType, result.markdown);

Key Source Files for Memory Buffer Processing

File	Purpose	Key Functions
src/lib.rs	Public API, re-exports, load_document_from_mem	detect_pdf_type_mem, extract_text_with_positions_mem, load_document_from_mem
src/detector.rs	Fast PDF classification	detect_pdf_type_mem, detect_pdf_type_mem_with_config
src/extractor/mod.rs	Text extraction orchestration	extract_text_with_positions_mem, font/content/layout pipeline
src/markdown/convert.rs	Markdown generation from extracted text	to_markdown

When to Use Memory Buffer Processing

Processing PDF files from a memory buffer is the optimal approach when:

• Receiving PDFs over HTTP — pass response.body or response.content directly without writing to disk
• Generating PDFs on-the-fly — processed documents from rendering services never touch storage
• Working with cached data — database BLOBs or in-memory caches (Redis, etc.) stream directly
• Serverless environments — ephemeral compute with limited writable storage (AWS Lambda, Cloud Functions)
• Security-sensitive workflows — keep confidential documents in RAM only, minimize attack surface

Summary

• pdf-inspector parses PDFs once via load_document_from_mem and reuses the lopdf::Document for all operations
• The *_mem functions (detect_pdf_type_mem, extract_text_with_positions_mem) accept &[u8] buffers with no filesystem dependency
• Available across Rust, Python, Node.js, and WebAssembly with identical semantics
• Pipeline includes structure repair, decryption, font analysis, content stream parsing, and layout detection—all in memory

Frequently Asked Questions

What memory buffer types does pdf-inspector accept?

The Rust API accepts any &[u8] slice. Language bindings map to native byte containers: Python bytes, Node.js Buffer/ Uint8Array, and WebAssembly Uint8Array. All internally convert to the same zero-copy view where possible.

Does memory buffer processing support password-protected PDFs?

Yes. Use load_document_from_mem_with_password (Rust) or equivalent binding methods to provide decryption credentials. The password-aware variant is called automatically when the PDF encryption dictionary is detected.

Is there a performance difference between file and memory buffer processing?

No meaningful difference—the same lopdf::Document parsing code executes either way. Memory buffers may be faster by eliminating disk I/O latency, especially for small PDFs or high-throughput scenarios.

Can I process multiple PDFs from memory in parallel?

Yes. Each load_document_from_mem call creates an independent Document instance. Clone your Vec<u8> or share Arc<[u8]> across threads/tasks, then invoke detection or extraction concurrently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}