# How pdf-inspector's Content Stream Operator State Machine Handles Tj, TJ, Td/TD, Tm, q, and Q

> Explore how pdf-inspector's state machine processes content stream operators like Tj, TJ, Td, Tm, q, and Q in Rust to extract structured text and bounding boxes.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: internals
- Published: 2026-09-02

---

**pdf‑inspector processes PDF content stream operators through a compact state machine in [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs) that maintains graphics and text state across six core operators to extract structured text items and bounding boxes.**

This article explains how the firecrawl/pdf‑inspector Rust library implements its operator-level content-stream parser. The state machine tracks matrices, fonts, and extraction flags while processing the six most common PDF text and graphics operators.

## Graphics State Operators: q and Q

The PDF specification uses **q** (*save graphics state*) and **Q** (*restore graphics state*) to bracket transformations. In pdf‑inspector, these manipulate a dedicated stack rather than matrices directly.

### q (Save Graphics State)

When the parser encounters `q`, it pushes a `SavedGraphicsState` snapshot onto `gstate_stack`. The snapshot captures:

- Current transformation matrix (**CTM**)
- Text and line matrices
- Line width, dash pattern, and miter limit
- Rendering intent and flatness tolerance
- Font reference, font size, and all text-state parameters

This implementation follows the PDF spec's stack semantics exactly. Multiple nested `q` operations create multiple stack entries.

### Q (Restore Graphics State)

The `Q` operator pops the most recent `SavedGraphicsState` and restores every captured field. The parser validates that a saved state exists; mismatched `Q` operators without prior `q` are handled gracefully by checking stack length before popping.

These operators enable complex page compositions where temporary clipping or transformations apply only to specific content regions.

## Text Positioning Operators: Td, TD, and Tm

Text positioning in PDF uses relative and absolute matrix operations. pdf‑inspector distinguishes `Td`/`TD` (relative moves) from `Tm` (absolute matrix setting).

### Td and TD (Translate Text Position)

Both operators move the text position by offset *(tx, ty)* relative to the **line matrix**:

- **`Td`**: Updates `line_matrix` by adding the scaled offset, then copies the result to `text_matrix`
- **`TD`**: Performs the same translation *and* sets `text_leading = -ty` (the negative of the y-offset)

The source code distinguishes these variants by operator name in the match arm, applying the additional leading assignment only for `TD`. This dual handling appears in the same code block for efficiency.

### Tm (Set Text Matrix)

The `Tm` operator accepts six numeric operands defining a full 3×2 transformation matrix. pdf‑inspector:

1. Reads all six operands in order [a b c d e f]
2. Writes them directly into `text_matrix`
3. Copies the same matrix to `line_matrix`

Copying to `line_matrix` ensures subsequent `Td`/`TD` operations calculate relative offsets from this new baseline. This matches the PDF specification's requirement that `Tm` establishes both the current text position and the line-start reference.

## Text Showing Operators: Tj and TJ

These operators render actual glyph content. The state machine performs substantial work: width calculation, invisible-text filtering, Unicode extraction, matrix advancement, and `TextItem` emission.

### Tj (Show String)

The `Tj` handler in [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs) executes five sequential steps:

1. **Compute string width** using `compute_string_width_ts` with current font metrics
2. **Check visibility**: if `text_rendering_mode == 3` (invisible) and `include_invisible` is false, skip glyph extraction but still advance the matrix
3. **Extract Unicode text** via `extract_text_from_operand` when visible or explicitly included
4. **Advance `text_matrix`** by the computed width, ensuring subsequent operators position correctly
5. **Emit `TextItem`** with position transformed through `rise_adjusted(&text_matrix, …)` composed with CTM

The `rise_adjusted` helper incorporates the `Ts` (text-rise) parameter for superscript/subscript positioning. The final matrix multiplication `multiply_matrices(..., &ctm)` converts from text space to user space coordinates.

Invisible text handling deserves special attention: when skipped, the parser sets `skipped_invisible = true` to signal downstream consumers that an OCR fallback may be appropriate.

### TJ (Show String with Positioning)

The `TJ` operator processes an array of alternating strings and numeric adjustments. This enables fine-grained glyph positioning for kerning and layout effects. pdf‑inspector's implementation:

1. **Calculate space threshold** from the font's space-width metric to detect word boundaries
2. **Iterate the operand array**, accumulating sub-strings separated by large negative offsets (column gaps)
3. **Create `TextItem` entries** for each sub-string, applying accumulated positioning adjustments
4. **Advance `text_matrix`** by the total array width after processing all elements
5. **Handle `ActualText` markup**: when `suppress_glyph_extraction` is set (accessibility replacement text), record the starting glyph matrix but emit no `TextItem`

The array-walking logic distinguishes positive adjustments (kerning) from large negative values (explicit gaps), using this to segment output into logical text runs.

## State Machine Architecture

Understanding the surrounding infrastructure clarifies how these six operators interrelate.

### Core State Variables

The parser maintains these fields while `in_text_block` is true (between `BT`/`ET`):

- `current_font`, `current_font_size`
- `char_spacing`, `word_spacing`
- `text_rise`, `text_leading`
- `text_matrix`, `line_matrix`
- `text_rendering_mode` (controls visibility)

Graphics state variables persist outside text blocks: `ctm` (current transformation matrix) and `gstate_stack`.

### Matrix Coordinate Flow

PDF coordinates transform through multiple spaces:

```

Glyph coordinates → text_matrix → CTM → user space → page coordinates

```

The `multiply_matrices` helper composes these transformations, while `rise_adjusted` applies the `Ts` parameter as a vertical translation before final composition.

### Extraction Flags and Modes

Two boolean flags modify operator behavior:

- **`include_invisible`**: When false, rendering mode 3 suppresses text extraction (but not matrix advancement)
- **`suppress_glyph_extraction`**: Set during `BDC`…`EMC` marked-content blocks with `ActualText` replacement

These flags enable accessibility-aware extraction that respects document intent while preserving positional information for layout analysis.

## Practical Usage Examples

### Command-Line Extraction

The `pdf2md` binary exercises all operator handling automatically:

```bash

# Extract structured text with position data

pdf2md --json sample.pdf > output.json

# Enable invisible text extraction

pdf2md --json --include-invisible sample.pdf > output.json

```

The JSON output contains `TextItem` objects whose `x` and `y` fields reflect the complete transformation chain including `Tm`, `Td`, and CTM effects.

### Library Integration

Direct Rust usage provides control over extraction options:

```rust
use pdf_inspector::{process_pdf_with_options, ProcessOptions};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = ProcessOptions::default();
    let (page_extractions, _, _, _) = process_pdf_with_options(
        "sample.pdf",
        &opts,
        &pdf_inspector::FontCMaps::default(),
        false,  // include_invisible
        &mut pdf_inspector::FontStyleCache::new(),
        &mut pdf_inspector::FormWalkBudget::default(),
    )?;

    for (items, _, _) in page_extractions {
        for item in items {
            println!("{:.2},{:.2}: {}", item.x, item.y, item.text);
        }
    }
    Ok(())
}

```

The `process_pdf_with_options` function invokes `extract_page_text_items`, which instantiates the content-stream state machine and populates the `TextItem` vector through the operator handling described above.

### Debugging Operator Processing

Instrument the state machine to trace specific operators:

```rust
// Add before the operator match in content_stream.rs
log::debug!(
    "op={} operands={:?} tm={:?} ctm={:?}",
    op.operator, op.operands, text_matrix, ctm
);

```

Run with `RUST_LOG=pdf_inspector::extractor::content_stream=debug` to observe step-by-step state transitions.

## Source File Reference

| File | Purpose |
|------|---------|
| [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs) | Operator state machine implementing `q`, `Q`, `Td`, `TD`, `Tm`, `Tj`, `TJ` handlers |
| [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) | Entry point calling `extract_page_text_items` per page |
| [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) | Public API exposing extraction functions |

## Summary

- **Graphics state stack**: `q` pushes snapshots to `gstate_stack`; `Q` pops and restores CTM and all subsidiary state
- **Relative positioning**: `Td`/`TD` offset the line matrix and synchronize text matrix; `TD` additionally sets leading
- **Absolute positioning**: `Tm` directly sets text matrix and copies to line matrix baseline
- **String showing**: `Tj` emits single `TextItem` with width-based matrix advancement; `TJ` processes arrays with positioning adjustments and gap detection
- **Invisible text**: Rendering mode 3 skips extraction (when configured) but preserves positioning for subsequent operators
- **Coordinate transformation**: All positions compose text matrix with CTM through `multiply_matrices`, with `text_rise` applied via `rise_adjusted`

## Frequently Asked Questions

### How does pdf-inspector handle nested q/Q operators without losing state?

The `gstate_stack: Vec<SavedGraphicsState>` supports arbitrary nesting. Each `q` pushes a complete snapshot; each `Q` pops the most recent. The vector structure naturally handles LIFO semantics, and the parser checks stack length before popping to tolerate malformed PDFs with unmatched `Q` operators.

### Why does Tm copy its matrix to the line matrix?

Per the PDF specification, `Tm` establishes both the current text position and the reference point for subsequent relative moves. Copying to `line_matrix` ensures that following `Td`/`TD` operators calculate offsets from this new baseline rather than the previous line's start position.

### What's the difference between invisible text (Tr=3) and ActualText suppression?

Invisible text (`text_rendering_mode == 3`) still represents rendered content that may be desired for extraction; pdf‑inspector provides `include_invisible` to control this. `ActualText` suppression occurs during marked-content blocks where the visible glyphs serve only as positioning placeholders for replacement text intended for accessibility tools—the parser records the position but skips glyph emission entirely.

### How does TJ detect word or column boundaries in the positioning array?

pdf‑inspector calculates a space-threshold from `font_widths` data. Negative numeric adjustments in the `TJ` array below this threshold indicate explicit gaps. The parser accumulates sub-strings separated by large negative offsets, emitting separate `TextItem` objects for each segment while advancing the matrix by the total array width.