How pdf-inspector's Content Stream Operator State Machine Handles Tj, TJ, Td/TD, Tm, q, and Q

pdf‑inspector processes PDF content stream operators through a compact state machine in src/extractor/content_stream.rs that maintains graphics and text state across six core operators to extract structured text items and bounding boxes.

This article explains how the firecrawl/pdf‑inspector Rust library implements its operator-level content-stream parser. The state machine tracks matrices, fonts, and extraction flags while processing the six most common PDF text and graphics operators.

Graphics State Operators: q and Q

The PDF specification uses q (save graphics state) and Q (restore graphics state) to bracket transformations. In pdf‑inspector, these manipulate a dedicated stack rather than matrices directly.

q (Save Graphics State)

When the parser encounters q, it pushes a SavedGraphicsState snapshot onto gstate_stack. The snapshot captures:

• Current transformation matrix (CTM)
• Text and line matrices
• Line width, dash pattern, and miter limit
• Rendering intent and flatness tolerance
• Font reference, font size, and all text-state parameters

This implementation follows the PDF spec's stack semantics exactly. Multiple nested q operations create multiple stack entries.

Q (Restore Graphics State)

The Q operator pops the most recent SavedGraphicsState and restores every captured field. The parser validates that a saved state exists; mismatched Q operators without prior q are handled gracefully by checking stack length before popping.

These operators enable complex page compositions where temporary clipping or transformations apply only to specific content regions.

Text Positioning Operators: Td, TD, and Tm

Text positioning in PDF uses relative and absolute matrix operations. pdf‑inspector distinguishes Td/TD (relative moves) from Tm (absolute matrix setting).

Td and TD (Translate Text Position)

Both operators move the text position by offset (tx, ty) relative to the line matrix:

• Td: Updates line_matrix by adding the scaled offset, then copies the result to text_matrix
• TD: Performs the same translation and sets text_leading = -ty (the negative of the y-offset)

The source code distinguishes these variants by operator name in the match arm, applying the additional leading assignment only for TD. This dual handling appears in the same code block for efficiency.

Tm (Set Text Matrix)

The Tm operator accepts six numeric operands defining a full 3×2 transformation matrix. pdf‑inspector:

1. Reads all six operands in order [a b c d e f]
2. Writes them directly into text_matrix
3. Copies the same matrix to line_matrix

Copying to line_matrix ensures subsequent Td/TD operations calculate relative offsets from this new baseline. This matches the PDF specification's requirement that Tm establishes both the current text position and the line-start reference.

Text Showing Operators: Tj and TJ

These operators render actual glyph content. The state machine performs substantial work: width calculation, invisible-text filtering, Unicode extraction, matrix advancement, and TextItem emission.

Tj (Show String)

The Tj handler in src/extractor/content_stream.rs executes five sequential steps:

1. Compute string width using compute_string_width_ts with current font metrics
2. Check visibility: if text_rendering_mode == 3 (invisible) and include_invisible is false, skip glyph extraction but still advance the matrix
3. Extract Unicode text via extract_text_from_operand when visible or explicitly included
4. Advance text_matrix by the computed width, ensuring subsequent operators position correctly
5. Emit TextItem with position transformed through rise_adjusted(&text_matrix, …) composed with CTM

The rise_adjusted helper incorporates the Ts (text-rise) parameter for superscript/subscript positioning. The final matrix multiplication multiply_matrices(..., &ctm) converts from text space to user space coordinates.

Invisible text handling deserves special attention: when skipped, the parser sets skipped_invisible = true to signal downstream consumers that an OCR fallback may be appropriate.

TJ (Show String with Positioning)

The TJ operator processes an array of alternating strings and numeric adjustments. This enables fine-grained glyph positioning for kerning and layout effects. pdf‑inspector's implementation:

1. Calculate space threshold from the font's space-width metric to detect word boundaries
2. Iterate the operand array, accumulating sub-strings separated by large negative offsets (column gaps)
3. Create TextItem entries for each sub-string, applying accumulated positioning adjustments
4. Advance text_matrix by the total array width after processing all elements
5. Handle ActualText markup: when suppress_glyph_extraction is set (accessibility replacement text), record the starting glyph matrix but emit no TextItem

The array-walking logic distinguishes positive adjustments (kerning) from large negative values (explicit gaps), using this to segment output into logical text runs.

State Machine Architecture

Understanding the surrounding infrastructure clarifies how these six operators interrelate.

Core State Variables

The parser maintains these fields while in_text_block is true (between BT/ET):

• current_font, current_font_size
• char_spacing, word_spacing
• text_rise, text_leading
• text_matrix, line_matrix
• text_rendering_mode (controls visibility)

Graphics state variables persist outside text blocks: ctm (current transformation matrix) and gstate_stack.

Matrix Coordinate Flow

PDF coordinates transform through multiple spaces:

Glyph coordinates → text_matrix → CTM → user space → page coordinates

The multiply_matrices helper composes these transformations, while rise_adjusted applies the Ts parameter as a vertical translation before final composition.

Extraction Flags and Modes

Two boolean flags modify operator behavior:

• include_invisible: When false, rendering mode 3 suppresses text extraction (but not matrix advancement)
• suppress_glyph_extraction: Set during BDC…EMC marked-content blocks with ActualText replacement

These flags enable accessibility-aware extraction that respects document intent while preserving positional information for layout analysis.

Practical Usage Examples

Command-Line Extraction

The pdf2md binary exercises all operator handling automatically:

# Extract structured text with position data

pdf2md --json sample.pdf > output.json

# Enable invisible text extraction

pdf2md --json --include-invisible sample.pdf > output.json

The JSON output contains TextItem objects whose x and y fields reflect the complete transformation chain including Tm, Td, and CTM effects.

Library Integration

Direct Rust usage provides control over extraction options:

use pdf_inspector::{process_pdf_with_options, ProcessOptions};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = ProcessOptions::default();
    let (page_extractions, _, _, _) = process_pdf_with_options(
        "sample.pdf",
        &opts,
        &pdf_inspector::FontCMaps::default(),
        false,  // include_invisible
        &mut pdf_inspector::FontStyleCache::new(),
        &mut pdf_inspector::FormWalkBudget::default(),
    )?;

    for (items, _, _) in page_extractions {
        for item in items {
            println!("{:.2},{:.2}: {}", item.x, item.y, item.text);
        }
    }
    Ok(())
}

The process_pdf_with_options function invokes extract_page_text_items, which instantiates the content-stream state machine and populates the TextItem vector through the operator handling described above.

Debugging Operator Processing

Instrument the state machine to trace specific operators:

// Add before the operator match in content_stream.rs
log::debug!(
    "op={} operands={:?} tm={:?} ctm={:?}",
    op.operator, op.operands, text_matrix, ctm
);

Run with RUST_LOG=pdf_inspector::extractor::content_stream=debug to observe step-by-step state transitions.

Source File Reference

File	Purpose
src/extractor/content_stream.rs	Operator state machine implementing q, Q, Td, TD, Tm, Tj, TJ handlers
src/extractor/mod.rs	Entry point calling extract_page_text_items per page
src/lib.rs	Public API exposing extraction functions

Summary

• Graphics state stack: q pushes snapshots to gstate_stack; Q pops and restores CTM and all subsidiary state
• Relative positioning: Td/TD offset the line matrix and synchronize text matrix; TD additionally sets leading
• Absolute positioning: Tm directly sets text matrix and copies to line matrix baseline
• String showing: Tj emits single TextItem with width-based matrix advancement; TJ processes arrays with positioning adjustments and gap detection
• Invisible text: Rendering mode 3 skips extraction (when configured) but preserves positioning for subsequent operators
• Coordinate transformation: All positions compose text matrix with CTM through multiply_matrices, with text_rise applied via rise_adjusted

Frequently Asked Questions

How does pdf-inspector handle nested q/Q operators without losing state?

The gstate_stack: Vec<SavedGraphicsState> supports arbitrary nesting. Each q pushes a complete snapshot; each Q pops the most recent. The vector structure naturally handles LIFO semantics, and the parser checks stack length before popping to tolerate malformed PDFs with unmatched Q operators.

Why does Tm copy its matrix to the line matrix?

Per the PDF specification, Tm establishes both the current text position and the reference point for subsequent relative moves. Copying to line_matrix ensures that following Td/TD operators calculate offsets from this new baseline rather than the previous line's start position.

What's the difference between invisible text (Tr=3) and ActualText suppression?

Invisible text (text_rendering_mode == 3) still represents rendered content that may be desired for extraction; pdf‑inspector provides include_invisible to control this. ActualText suppression occurs during marked-content blocks where the visible glyphs serve only as positioning placeholders for replacement text intended for accessibility tools—the parser records the position but skips glyph emission entirely.

How does TJ detect word or column boundaries in the positioning array?

pdf‑inspector calculates a space-threshold from font_widths data. Negative numeric adjustments in the TJ array below this threshold indicate explicit gaps. The parser accumulates sub-strings separated by large negative offsets, emitting separate TextItem objects for each segment while advancing the matrix by the total array width.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}