How to Process PDF Content from Memory with pdf-inspector

Yes, pdf-inspector provides dedicated in-memory functions process_pdf_mem and process_pdf_mem_with_options that accept raw byte buffers (&[u8]) to parse PDFs without writing to disk, returning the same structured PdfProcessResult as the file-based API.

The firecrawl/pdf-inspector repository includes a fully-featured in-memory API designed for scenarios where PDF data exists as bytes rather than files. This capability is essential for web services, serverless functions, and streaming pipelines where writing temporary files introduces unnecessary latency or security concerns.

Rust In-Memory API

The Rust core library exposes two primary entry points for processing PDF content from memory. Both functions reside in src/lib.rs and delegate to the same extraction pipeline as the file-based methods.

Basic Buffer Processing

For straightforward extraction with default options, use process_pdf_mem. Implemented at line 298 in src/lib.rs, this function accepts a &[u8] slice and returns a Result<PdfProcessResult, PdfInspectorError>.

use pdf_inspector::process_pdf_mem;

fn extract_from_memory(pdf_bytes: &[u8]) {
    match process_pdf_mem(pdf_bytes) {
        Ok(result) => println!("Markdown output:\n{}", result.markdown),
        Err(e) => eprintln!("Failed to parse PDF: {}", e),
    }
}

Advanced Options and Configuration

When you need custom detection modes, specific page selection, or structured JSON output, use process_pdf_mem_with_options. Located at line 310 in src/lib.rs, this function pairs a byte buffer with a PdfOptions struct.

use pdf_inspector::{process_pdf_mem_with_options, PdfOptions, ProcessMode};

fn detect_from_memory(pdf_bytes: &[u8]) {
    let opts = PdfOptions::new()
        .mode(ProcessMode::DetectOnly)  // Run only PDF type detection
        .json(true);                     // Request structured JSON output

    match process_pdf_mem_with_options(pdf_bytes, opts) {
        Ok(result) => println!("JSON result:\n{}", result.json.unwrap()),
        Err(e) => eprintln!("Detection error: {}", e),
    }
}

Both functions preserve the full extraction capabilities available in file-based processing, including table detection, layout heuristics, and Markdown conversion handled in src/markdown/convert.rs.

Python Byte Buffer Support

The Python bindings expose in-memory processing through process_pdf_bytes, implemented in src/python.rs at line 496. This wrapper forwards the Python bytes object to the Rust core via process_pdf_mem_with_options.

import pdf_inspector

def extract_pdf(data: bytes):
    # Default extraction (equivalent to process_pdf_mem)

    result = pdf_inspector.process_pdf_bytes(data)
    print(result.markdown)

def detect_pdf(data: bytes):
    # Detect-only mode with JSON output

    result = pdf_inspector.process_pdf_bytes(
        data,
        pages=None,
        options=pdf_inspector.PdfOptions(detect_only=True, json=True)
    )
    print(result.json)

Core Implementation Pipeline

When processing PDF content from memory, the library routes data through the same architecture as file-based operations:

• src/extractor/mod.rs – Orchestrates the extraction pipeline, invoked identically by both memory and file entry points
• src/detector.rs – Executes PDF-type detection logic when detect_only mode is enabled
• src/markdown/convert.rs – Transforms internal text structures into final Markdown or JSON format

Because the byte buffer API skips disk I/O and passes the &[u8] slice directly to the parser in src/extractor/mod.rs, performance characteristics remain identical to file processing while eliminating filesystem overhead.

When to Use In-Memory Processing

Processing PDF content from memory offers distinct advantages in specific architectures:

• Serverless environments – AWS Lambda and Cloud Functions where ephemeral storage is limited or costly
• Web uploads – Handling multipart/form-data directly without saving to disk
• Stream processing – Consuming PDFs from message queues or network streams
• Security-sensitive applications – Avoiding temporary file creation for confidential documents

Summary

• process_pdf_mem in src/lib.rs (line 298) provides the simplest in-memory interface using default options
• process_pdf_mem_with_options in src/lib.rs (line 310) enables full configuration of detection modes and output formats
• Python users call process_pdf_bytes from src/python.rs (line 496) to access the same byte buffer functionality
• The extraction pipeline in src/extractor/mod.rs processes memory buffers and files identically, ensuring consistent Markdown/JSON output quality
• Both Rust and Python APIs support detect-only mode for lightweight PDF classification without full text extraction

Frequently Asked Questions

Can pdf-inspector handle large PDFs entirely in memory?

Yes, pdf-inspector processes the byte buffer as a &[u8] slice, but memory constraints depend on your runtime environment. For extremely large documents, ensure your host has sufficient RAM to hold both the input buffer and the extraction results, as the library does not currently support chunked streaming of the input buffer.

Does the in-memory API support the same output formats as the file API?

Absolutely. Whether you use process_pdf_mem or process_pdf, the resulting PdfProcessResult contains identical fields: markdown for text output and json for structured data. The conversion logic in src/markdown/convert.rs executes the same code path regardless of input source.

How do I process PDF bytes from an HTTP request in Python?

Pass the request body directly as bytes to process_pdf_bytes. Since the function accepts a standard Python bytes object, frameworks like FastAPI or Flask can stream the request content into pdf-inspector without intermediate file storage:

result = pdf_inspector.process_pdf_bytes(request.content, options=pdf_inspector.PdfOptions(json=True))

Is there a performance difference between file and memory processing?

The parsing engine in src/extractor/mod.rs executes identical logic for both input methods. The only performance difference is the elimination of disk I/O when using the memory API, often resulting in faster processing for network-resident or ephemeral PDF data.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}