How pdf-inspector Handles PDF Operator States: A Deep Dive into the Rust State Machine

pdf-inspector tracks PDF operator states using a minimal Rust state machine in src/extractor/content_stream.rs that mirrors the PDF graphics and text model, with stack-based save/restore operations via q/Q operators and matrix transformations via cm.

The firecrawl/pdf-inspector crate parses PDF content streams to extract structured text while preserving layout. Understanding how it handles PDF operator states reveals why it can accurately position glyphs, detect underlines, and respect nested transformations. This article breaks down the state machine implementation, from graphics stack management to text matrix calculations.

The Core State Machine Architecture

pdf-inspector's extractor follows a classic interpreter pattern: iterate over decoded operations, update mutable state, and emit TextItem objects when text is rendered. The implementation lives primarily in src/extractor/content_stream.rs, with roughly 200 lines dedicated to operator dispatch.

The state machine manages several distinct but interacting subsystems:

• Graphics state — transformation matrices, line width, clipping paths
• Text state — current font, size, spacing, rendering mode
• Text positioning — text matrix (Tm) and line matrix (Tlm)
• Marked content — accessibility tags and logical structure

Each subsystem responds to specific PDF operators, and their interaction determines where each glyph appears in the final output.

Graphics State: The q/Q Stack Pattern

PDF graphics state uses a stack-based save/restore mechanism. The q operator pushes the current state; Q pops and restores it.

In src/extractor/content_stream.rs (lines 39–53), this is implemented with a Vec<SavedGraphicsState>:

if op.operator == "q" {
    gstate_stack.push(SavedGraphicsState {
        ctm,
        line_width,
        text_rendering_mode,
        // ... other fields
    });
} else if op.operator == "Q" {
    if let Some(saved) = gstate_stack.pop() {
        ctm = saved.ctm;
        line_width = saved.line_width;
        text_rendering_mode = saved.text_rendering_mode;
        // ... restore all fields
    }
}

This pattern is critical for correctness: PDF generators frequently wrap transformations in q/Q pairs to isolate changes. Without proper stack handling, a rotation or scale applied to one element would leak into subsequent elements.

The SavedGraphicsState struct captures a complete snapshot, ensuring that any graphics state change—no matter how deep in the nesting—is properly scoped.

Current Transformation Matrix (CTM) Handling

The CTM maps user space coordinates to device space. It's a 6-element affine transformation matrix that every coordinate passes through.

The cm operator concatenates a new matrix onto the existing CTM. In content_stream.rs (lines 67–79):

let new_matrix = [
    op.operands[0].as_f64_or_0(),
    op.operands[1].as_f64_or_0(),
    op.operands[2].as_f64_or_0(),
    op.operands[3].as_f64_or_0(),
    op.operands[4].as_f64_or_0(),
    op.operands[5].as_f64_or_0(),
];
ctm = multiply_matrices(&new_matrix, &ctm);

Note the matrix multiplication order: new_matrix * ctm, not ctm * new_matrix. This follows the PDF specification where new transformations apply "after" existing ones in the pipeline.

The multiply_matrices helper performs standard 3×3 matrix multiplication on the 2D affine representation. Incorrect order here would cause translations, rotations, and scales to compose backwards—producing misplaced text in documents with complex transformations.

Line Width and Stroke State

For detecting underlines and other decorative lines, pdf-inspector tracks the current line width via the w operator:

"w" => {
    line_width = op.operands[0].as_f64_or_0();
}

This simple assignment (lines 81–85 in content_stream.rs) enables downstream layout analysis. When the extractor later encounters a rectangle or line-drawing operator, it can compare the stroke width against font metrics to identify underlines versus borders.

Text Block Management with BT/ET

PDF text operations must occur inside text objects bounded by BT (begin text) and ET (end text) operators. The state machine tracks this with a boolean flag and resets positioning state on entry:

"BT" => {
    in_text_block = true;
    text_matrix = [1.0, 0.0, 0.0, 1.0, 0.0, 0.0];
    line_matrix = [1.0, 0.0, 0.0, 1.0, 0.0, 0.0];
}
"ET" => {
    in_text_block = false;
}

This reset behavior (lines 86–96) is spec-mandated: each text object starts with identity matrices, preventing text positioning from leaking across unrelated blocks.

Text State: Font, Size, and Rendering Mode

Inside a text block, the text state controls how glyphs are rendered. Key operators handled in content_stream.rs (lines 98–124):

Operator	Purpose	Variable Updated
Tf	Set font and size	current_font, font_size
TL	Set text leading	text_leading
Tr	Set rendering mode	text_rendering_mode
Tc	Set character spacing	char_spacing
Tw	Set word spacing	word_spacing
Ts	Set text rise	rise

The rendering mode (Tr) deserves special attention. Mode 3 means "neither fill nor stroke"—invisible text often used for OCR layer alignment. pdf-inspector checks this value when emitting TextItems, either skipping invisible glyphs or recording them for retry logic when no visible text is found.

Text Positioning: Tm, Tlm, and Glyph Placement

The heart of accurate text extraction lies in text matrix management. PDF uses two matrices:

• Text matrix (Tm) — defines the coordinate system for the current glyph sequence
• Line matrix (Tlm) — remembers where the current line started, used for line-breaking operators

Operators that modify positioning (lines 124–170):

• Td tx ty — move to start of next line, offset by (tx, ty) from line matrix start
• TD tx ty — same as Td but also sets text leading to ty
• Tm a b c d e f — absolutely set the text matrix (used for precise positioning)
• T* — move to next line using current text leading

After each show-text operator (Tj, TJ, ', "), the matrix advances by the computed string width. This auto-advancement is what makes PDF text extraction challenging: you must measure each string, account for spacing parameters, and update the matrix before the next operation.

Show-Text Operators: From Bytes to TextItems

The final stage converts PDF string operands into positioned TextItem structures. In content_stream.rs (lines 75–115), the show-text handler:

1. Decodes the operand bytes using the current font's encoding
2. Computes glyph widths via compute_string_width_ts
3. Adjusts for text rise (rise_adjusted translation)
4. Transforms through CTM to get device-space coordinates
5. Creates a TextItem with bounds, content, and font metadata

Invisible text (rendering mode 3) receives special handling: the extractor may skip it or flag it for accessibility processing, depending on configuration.

Marked Content and Logical Structure

PDF accessibility relies on marked content operators that tag regions with MCIDs (marked content IDs) and ActualText for alternative representations.

The state machine maintains a marked_content_stack (lines 124–136):

"BMC" | "BDC" => {
    let tag = op.operands[0].as_name_str().unwrap_or_default();
    let mcid = /* extract from properties */;
    marked_content_stack.push(MarkedContentEntry { tag: tag.to_string(), mcid });
}
"EMC" => {
    marked_content_stack.pop();
}

Each emitted TextItem captures the innermost MCID, enabling later reconstruction of document structure for screen readers and semantic analysis.

XObject Integration: Images and Forms

The Do operator draws XObjects—external objects including images and form XObjects (nested content streams). pdf-inspector handles these in content_stream.rs (lines 165–210) and src/extractor/xobjects.rs.

For images, it computes a bounding box from the current CTM and emits a synthetic TextItem:

let image_bbox = image_bbox_from_ctm(&ctm);
items.push(TextItem::ImagePlaceholder {
    rect: image_bbox,
    // ...
});

For form XObjects, it recursively calls extract_form_xobject_text, passing the current CTM as the inherited transformation. This nesting is essential: form XObjects have their own content streams but inherit and concatenate with the parent's coordinate system.

The recursion is bounded by a FormWalkBudget to prevent infinite loops from malformed PDFs with circular form references.

Practical API Usage

The state machine is exercised through two primary interfaces. For standard extraction:

use pdf_inspector::{process_pdf_with_options, ProcessOptions};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = ProcessOptions {
        json: true,
        ..Default::default()
    };
    let (json, _warnings) = process_pdf_with_options("sample.pdf", &opts)?;
    println!("{}", json);
    Ok(())
}

For custom pipelines that need direct state machine access:

use pdf_inspector::{extractor::content_stream::extract_page_text_items, FontCMaps};

fn custom_extract(doc: &lopdf::Document, page_id: lopdf::ObjectId) {
    let font_cmaps = FontCMaps::default();
    let mut style_cache = pdf_inspector::extractor::fonts::FontStyleCache::new();
    let mut form_budget = pdf_inspector::extractor::xobjects::FormWalkBudget::default();

    let (page_extraction, _, _, _) = extract_page_text_items(
        doc,
        page_id,
        1,
        &font_cmaps,
        false,
        &mut style_cache,
        &mut form_budget,
    ).unwrap();

    for item in page_extraction.0 {
        println!("{:?}", item);
    }
}

Both paths ultimately invoke the operator state machinery in content_stream.rs, ensuring consistent handling of graphics state, text positioning, and nested structures.

Key Files and Responsibilities

File	Role in PDF Operator State Handling
src/extractor/content_stream.rs	Core state machine; implements operator dispatch, matrix math, stack management, and TextItem emission
src/extractor/fonts.rs	Font encoding tables and glyph width resolution; consumed by show-text operators
src/extractor/xobjects.rs	XObject lookup and recursive form processing; manages FormWalkBudget
src/extractor/mod.rs	Pipeline orchestration; calls page extraction repeatedly
src/detector.rs	PDF type detection; gates whether operator state machine runs

Summary

• pdf-inspector handles PDF operator states through a Rust state machine in src/extractor/content_stream.rs that faithfully implements the PDF graphics and text models
• Graphics state uses a Vec<SavedGraphicsState> stack manipulated by q (push) and Q (pop) operators, ensuringscoped transformations
• The CTM is updated via matrix multiplication in the cm operator, with correct order (new * existing) per the PDF specification
• Text state variables—font, size, spacing, rendering mode—are set by dedicated operators and consumed during show-text processing
• Text positioning relies on text matrix (Tm) and line matrix (Tlm) updates via Td, TD, Tm, and T* operators, with automatic advancement after each string
• Marked content and XObjects extend the state machine with stack-based tagging and recursive form processing, both respecting inherited transformations

Frequently Asked Questions

How does pdf-inspector handle nested transformations in PDFs?

pdf-inspector uses the q (save) and Q (restore) operators to manage a stack of SavedGraphicsState structs. When q is encountered, the current CTM and other graphics state are cloned and pushed; Q pops and restores them. This stack-based approach correctly scopes transformations regardless of nesting depth, as implemented in src/extractor/content_stream.rs lines 39–53.

What determines the final position of extracted text?

Final text position results from cascading transformations: the CTM (set by cm and inherited from parent contexts) multiplied by the text matrix (set by Tm or updated by Td/TD/T*), plus text rise adjustment. The show-text operators apply this combined transformation to each glyph's origin, producing device-space coordinates stored in TextItem.x and TextItem.y.

How does pdf-inspector detect and handle invisible text?

The Tr (set rendering mode) operator sets text_rendering_mode. Mode 3 means "invisible" (neither fill nor stroke). The show-text handler in content_stream.rs checks this value and either skips invisible glyphs or records them for retry logic, depending on whether the document contains visible alternatives. This handles common OCR-invisible text patterns.

Can pdf-inspector extract text from PDF forms and annotations?

Yes, via the Do operator handler. When a form XObject is referenced, extract_form_xobject_text is called recursively with the current CTM as the inherited transformation. This processes the nested content stream while preserving positioning context. The recursion is bounded by FormWalkBudget to prevent infinite loops from malformed circular references.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}