How pdf-inspector Extracts Form XObject Text and Image Placeholders from PDF Content Streams

pdf-inspector processes PDF pages as a content-stream state machine, using the Do operator to detect XObject references and recursively extracting Form XObject text while creating lightweight placeholders for Image XObjects.

Form XObject and image extraction is a core capability of the pdf-inspector library. The crate treats every PDF page as a traversable sequence of graphics operators, with special handling for the Do operator—the PDF instruction that invokes external objects. This article breaks down how the library distinguishes Form XObjects from Image XObjects, applies transformation matrices recursively, and enforces safety budgets to prevent denial-of-service attacks.

The extraction pipeline lives primarily in src/extractor/xobjects.rs and src/extractor/content_stream.rs, with public APIs exposed through src/lib.rs.

How the Do Operator Triggers XObject Extraction

Every PDF page contains a content stream—a byte sequence of operators and operands that describe what to render. The Do operator (PDF 1.7 spec §4.7) takes a name operand that references an entry in the page's /Resources dictionary under /XObject.

In src/extractor/content_stream.rs, the state machine detects this operator and delegates to type-specific handlers:

• Form XObjects → recursive stream expansion via extract_form_xobject
• Image XObjects → placeholder generation via extract_image_xobject

The lookup occurs at runtime against the page's resource inheritance chain, ensuring XObjects defined in parent pages or document catalogs are correctly resolved.

Extracting Text from Nested Form XObjects

Form XObjects are self-contained content streams with their own resource dictionaries and optional transformation matrices. They enable PDF creators to reuse graphical elements—logos, headers, template blocks—across multiple pages.

Step-by-Step Form XObject Processing

extract_form_xobject in src/extractor/xobjects.rs (around line 181) implements the following sequence:

1. Stream retrieval — get_form_xobject_stream (line 122) decodes the XObject's bytes, handling FlateDecode, LZWDecode, and other PDF filter chains.

2. Matrix application — If the Form XObject specifies a /Matrix array, apply_form_matrix (line 305) concatenates it with the current Current Transformation Matrix (CTM). This ensures text coordinates nest correctly into page space.

3. Nested parsing — The decoded bytes wrap in a new ContentStream instance, reusing the same parser logic as top-level pages. The library walks operators including:

   • Tj, TJ — show text strings
   • ', " — show text with spacing
   • Tm, Td, TD, T* — text positioning

4. Item merging — Extracted TextItem objects merge into the parent page's item list, preserving reading order.

Recursion Budget and Attack Prevention

PDFs can embed Form XObjects that reference other Form XObjects—either legitimately for complex templates or maliciously for billion laughs-style denial of service. pdf-inspector enforces strict limits in src/extractor/xobjects.rs (lines 19–30):

/// Maximum number of Form XObject invocations per page.
pub const MAX_FORM_XOBJECT_INVOKE: usize = 100;

/// Maximum total operations across all Form XObject expansions.
pub const MAX_FORM_XOBJECT_OPS: usize = 100_000;

Each invocation increments a per-extraction counter. Exceeding either limit truncates recursion and emits a warning (logged at line 1292 of content_stream.rs). This design surfaces the "Form XObject expansion truncated" diagnostic that operators can monitor.

Creating Image Placeholders Without Rasterization

Image XObjects require different handling. Rather than decode pixel data—which would consume memory and CPU—pdf-inspector records position and bounding box only.

Placeholder Generation Flow

extract_image_xobject in src/extractor/xobjects.rs (around line 413) executes:

1. Resolves the image XObject reference from /Resources
2. Constructs a TextItem with kind: ItemKind::ImagePlaceholder
3. Records the image's rectangle via PdfRect
4. Sets a sentinel string—typically "[IMG]"—for downstream processing

No image filters run. No bitmap buffers allocate. The placeholder preserves the visual flow of the original PDF without the extraction cost.

Markdown Post-Processing

The src/markdown/postprocess.rs stage transforms image placeholders into standard Markdown syntax:

<!-- Placeholder becomes: -->
![](image-url-placeholder)

This separation of concerns—extraction versus presentation—lets callers substitute actual image URLs or base64 data in later pipeline stages.

Complete Extraction Example

The following Rust program demonstrates both Form XObject text extraction and image placeholder handling:

use pdf_inspector::{process_pdf_with_options, ExtractionOptions};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Default extraction options include Form XObject budgets.
    let opts = ExtractionOptions::default();

    // Extract document structure.
    let doc = process_pdf_with_options("sample.pdf", opts)?;

    for (page_idx, page) in doc.pages.iter().enumerate() {
        println!("--- Page {} ---", page_idx + 1);
        
        for item in &page.items {
            match &item.kind {
                // Text from page or nested Form XObjects.
                pdf_inspector::types::ItemKind::Text(txt) => {
                    println!("text: {}", txt);
                }
                // Image placeholder—position preserved, no pixel data.
                pdf_inspector::types::ItemKind::ImagePlaceholder => {
                    println!("image at {}", item.rect);
                }
                _ => {}
            }
        }
    }
    Ok(())
}

The ExtractionOptions::default() configuration applies the standard MAX_FORM_XOBJECT_INVOKE and MAX_FORM_XOBJECT_OPS limits. Applications processing trusted documents can adjust these thresholds via the options struct.

Architectural Comparison: Form vs. Image XObjects

Aspect	Form XObject	Image XObject
Content type	Nested content stream (text, graphics, other XObjects)	Pixel data or sampled image
Recursion	Deep—may contain other XObjects	None—terminal leaf
Extraction output	Flattened TextItem sequence	Single ImagePlaceholder item
Matrix handling	Required—concatenated with parent CTM	Optional—position only
Budget enforcement	MAX_FORM_XOBJECT_INVOKE, MAX_FORM_XOBJECT_OPS	None—constant-time handling
Resource inheritance	Full /Resources dictionary	Minimal—subtype and rectangle

Key Source Files and Functions

Path	Responsibility
src/extractor/xobjects.rs	Core XObject logic: get_form_xobject_stream, extract_form_xobject, extract_image_xobject, apply_form_matrix, budget constants
src/extractor/content_stream.rs	State machine walking page/Form streams; Do operator detection; warning emission at line 1292
src/lib.rs	Public API entry point process_pdf_with_options; extraction context setup
src/markdown/postprocess.rs	Placeholder-to-Markdown transformation
src/types.rs	ItemKind enum variants: Text, ImagePlaceholder

Summary

• pdf-inspector extracts Form XObject text by recursively parsing nested content streams, applying transformation matrices, and merging results into the parent page's item list.

• Safety budgets (MAX_FORM_XOBJECT_INVOKE, MAX_FORM_XOBJECT_OPS) prevent resource exhaustion from malicious or degenerate PDF structures.

• Image XObjects generate lightweight placeholders recording position and bounding box—no pixel decoding occurs during extraction.

• The Do operator serves as the universal hook for both object types, with dispatch occurring in src/extractor/content_stream.rs and implementation details in src/extractor/xobjects.rs.

Frequently Asked Questions

How does pdf-inspector handle deeply nested Form XObjects?

The library enforces hard limits on both invocation count and total operations. MAX_FORM_XOBJECT_INVOKE caps the number of distinct Form XObjects expanded per page, while MAX_FORM_XOBJECT_OPS limits aggregate operator processing across all expansions. Exceeding either threshold truncates processing and logs a warning.

Can I extract actual image pixel data instead of placeholders?

Not through the current extraction API. pdf-inspector deliberately avoids image decoding to minimize resource consumption. Applications requiring pixel access must implement separate image extraction using the resolved XObject reference and PDF image parameters (width, height, color space, filters).

Why are image placeholders represented as TextItem structures?

Unification simplifies the downstream processing pipeline. TextItem carries position (PdfRect), content string, and kind discriminator—sufficient for both text flow preservation and later Markdown transformation. This design avoids parallel collection types while maintaining type safety through the ItemKind enum.

What PDF filters does the Form XObject stream decoder support?

get_form_xobject_stream in src/extractor/xobjects.rs handles standard PDF filters including FlateDecode (zlib/Deflate), LZWDecode, ASCIIHexDecode, and ASCII85Decode. Filter chains specified in the XObject's stream dictionary apply sequentially during byte retrieval.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}