# How pdf-inspector Makes CMap Decisions for Font Width and Encoding

> Discover how pdf-inspector decides CMap for font width and encoding by sampling, scoring Unicode quality, and caching the best choice after analyzing 240 bytes of data.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: internals
- Published: 2026-09-02

---

**The font width and encoding module in pdf-inspector makes CMap decisions by sampling text from both the primary ToUnicode stream and a remapped TrueType-based fallback, scoring Unicode quality, and caching the superior choice once 240 bytes of source data have been analyzed.**

PDF fonts encode character data as raw byte sequences that require a **CMap** (character-to-Unicode map) to produce readable text. According to the firecrawl/pdf-inspector source code, each font may expose two competing maps—a primary `ToUnicode` stream and a remapped fallback built from TrueType `cmap` tables or glyph names. The extractor must decide which map to trust, and it does so through a statistical sampling heuristic implemented in [`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs).

## The CMapDecisionCache Architecture

The core decision mechanism lives in the `CMapDecisionCache` struct:

```rust
pub(crate) struct CMapDecisionCache {
    decisions: HashMap<u32, CMapDecision>,
}

```

- Keys are **PDF object numbers** of the `ToUnicode` stream
- Values store accumulated text samples and the final `CMapChoice` (`Primary` or `Remapped`)

This caching strategy ensures the heuristic runs **once per font object**, making extraction fast even on large documents with thousands of pages.

## The consider Method: Sampling and Scoring

During page processing, the extractor calls `consider` with parallel text output from both maps:

```rust
decisions.consider(
    obj_num,
    &primary_text,   // text from the primary ToUnicode CMap
    &remapped_text,  // text from the remapped TrueType CMap
    slice_len,       // source byte count this slice represents
);

```

The `consider` method executes a four-step decision workflow:

1. **Accumulates** text fragments into `primary_sample` and `remapped_sample` fields, tracking total `sample_bytes`
2. **Pauses** when `SAMPLE_TARGET_BYTES` (240 bytes) is reached—sufficient for statistical significance without parsing entire documents
3. **Scores** both samples using `score_text`, which rewards legible Unicode (letters, numbers, common punctuation) and penalizes replacement characters (`U+FFFD`), control symbols, and nonsensical sequences
4. **Selects** based on a 5-point margin threshold:

```rust
let score_primary = score_text(&entry.primary_sample);
let score_remap   = score_text(&entry.remapped_sample);

entry.choice = if score_remap > score_primary + 5 {
    Some(CMapChoice::Remapped)
} else {
    Some(CMapChoice::Primary)
};

```

The primary CMap wins ties or close calls—the remapped alternative must prove substantially better.

## Applying Cached Decisions During Extraction

At extraction time (approximately lines 2160–2240 of [`fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/fonts.rs)), the code retrieves the stored choice:

```rust
let cmap_choice = decisions
    .get_choice(font_tounicode_refs.get(current_font).copied().unwrap_or(0));

```

The extractor then branches:

- **Primary** → use the original `ToUnicode` CMap directly
- **Remapped** → use the TrueType `cmap` or glyph-name reconstruction

## Practical Example: Detecting Corrupted ToUnicode Maps

```rust
use pdf_inspector::extractor::fonts::{CMapDecisionCache, CMapChoice};

fn demo_decision() {
    // Simulate garbled vs. clean output for the same font
    let obj_num = 42;
    let primary   = "¶©";  // mojibake from broken ToUnicode
    let remapped  = "¶©";       // correct Unicode via TrueType cmap
    let mut cache = CMapDecisionCache::new();

    // Accumulate samples until threshold reached
    for _ in 0..3 {
        cache.consider(obj_num, primary, remapped, 100);
    }

    match cache.get_choice(obj_num) {
        Some(CMapChoice::Primary)   => println!("Trust original ToUnicode"),
        Some(CMapChoice::Remapped)  => println!("Trust remapped TrueType CMap"),
        None                        => println!("Insufficient sample data"),
    }
}

```

Output:

```

Trust remapped TrueType CMap

```

The remapped sample scores higher due to valid Unicode punctuation versus the primary sample's garbled byte sequences.

## Why This CMap Decision Strategy Matters

- **Subset PDFs** often ship `ToUnicode` tables referencing original GIDs that become invalid after font subsetting; the remapped CMap corrects GID→Unicode relationships
- **Lazy evaluation** via 240-byte sampling avoids full-document parsing overhead
- **Per-object caching** provides O(1) lookup for subsequent pages using the same font

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs) | `CMapDecisionCache` implementation and heuristic scoring integration |
| [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) | ToUnicode stream parsing and TrueType fallback CMap construction |
| [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) | `score_text` function for Unicode quality assessment |
| [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) | Orchestration wiring caching into the text-decoding pipeline |

## Summary

- **CMap decisions in pdf-inspector** are made once per font object using a sampling-based quality heuristic
- The **240-byte threshold** balances statistical reliability against extraction performance
- A **5-point scoring margin** prevents switching to remapped maps without substantial evidence
- The **per-object cache** eliminates redundant analysis across multi-page documents

## Frequently Asked Questions

### What triggers pdf-inspector to choose the remapped CMap over the primary ToUnicode stream?

The remapped CMap is selected when its accumulated text sample scores more than 5 points higher than the primary sample on the `score_text` heuristic. This margin ensures the TrueType-based reconstruction is substantially more reliable, not merely marginally better.

### How much PDF content does pdf-inspector need to analyze before making a CMap decision?

The extractor requires **240 bytes** of source data (`SAMPLE_TARGET_BYTES`) before finalizing a choice. This threshold, implemented in [`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs), provides sufficient statistical signal for Unicode quality assessment without requiring full document parsing.

### Can pdf-inspector's CMap decision change after the initial 240-byte sample?

No. Once `consider` accumulates enough data and stores a `CMapChoice` in the cache, all subsequent calls to `get_choice` for that object number return the cached result immediately. This design guarantees consistent text decoding across all pages using the same font.