How pdf-inspector Makes CMap Decisions for Font Width and Encoding

The font width and encoding module in pdf-inspector makes CMap decisions by sampling text from both the primary ToUnicode stream and a remapped TrueType-based fallback, scoring Unicode quality, and caching the superior choice once 240 bytes of source data have been analyzed.

PDF fonts encode character data as raw byte sequences that require a CMap (character-to-Unicode map) to produce readable text. According to the firecrawl/pdf-inspector source code, each font may expose two competing maps—a primary ToUnicode stream and a remapped fallback built from TrueType cmap tables or glyph names. The extractor must decide which map to trust, and it does so through a statistical sampling heuristic implemented in src/extractor/fonts.rs.

The CMapDecisionCache Architecture

The core decision mechanism lives in the CMapDecisionCache struct:

pub(crate) struct CMapDecisionCache {
    decisions: HashMap<u32, CMapDecision>,
}

• Keys are PDF object numbers of the ToUnicode stream
• Values store accumulated text samples and the final CMapChoice (Primary or Remapped)

This caching strategy ensures the heuristic runs once per font object, making extraction fast even on large documents with thousands of pages.

The consider Method: Sampling and Scoring

During page processing, the extractor calls consider with parallel text output from both maps:

decisions.consider(
    obj_num,
    &primary_text,   // text from the primary ToUnicode CMap
    &remapped_text,  // text from the remapped TrueType CMap
    slice_len,       // source byte count this slice represents
);

The consider method executes a four-step decision workflow:

1. Accumulates text fragments into primary_sample and remapped_sample fields, tracking total sample_bytes
2. Pauses when SAMPLE_TARGET_BYTES (240 bytes) is reached—sufficient for statistical significance without parsing entire documents
3. Scores both samples using score_text, which rewards legible Unicode (letters, numbers, common punctuation) and penalizes replacement characters (U+FFFD), control symbols, and nonsensical sequences
4. Selects based on a 5-point margin threshold:

let score_primary = score_text(&entry.primary_sample);
let score_remap   = score_text(&entry.remapped_sample);

entry.choice = if score_remap > score_primary + 5 {
    Some(CMapChoice::Remapped)
} else {
    Some(CMapChoice::Primary)
};

The primary CMap wins ties or close calls—the remapped alternative must prove substantially better.

Applying Cached Decisions During Extraction

At extraction time (approximately lines 2160–2240 of fonts.rs), the code retrieves the stored choice:

let cmap_choice = decisions
    .get_choice(font_tounicode_refs.get(current_font).copied().unwrap_or(0));

The extractor then branches:

• Primary → use the original ToUnicode CMap directly
• Remapped → use the TrueType cmap or glyph-name reconstruction

Practical Example: Detecting Corrupted ToUnicode Maps

use pdf_inspector::extractor::fonts::{CMapDecisionCache, CMapChoice};

fn demo_decision() {
    // Simulate garbled vs. clean output for the same font
    let obj_num = 42;
    let primary   = "¶©";  // mojibake from broken ToUnicode
    let remapped  = "¶©";       // correct Unicode via TrueType cmap
    let mut cache = CMapDecisionCache::new();

    // Accumulate samples until threshold reached
    for _ in 0..3 {
        cache.consider(obj_num, primary, remapped, 100);
    }

    match cache.get_choice(obj_num) {
        Some(CMapChoice::Primary)   => println!("Trust original ToUnicode"),
        Some(CMapChoice::Remapped)  => println!("Trust remapped TrueType CMap"),
        None                        => println!("Insufficient sample data"),
    }
}

Output:

Trust remapped TrueType CMap

The remapped sample scores higher due to valid Unicode punctuation versus the primary sample's garbled byte sequences.

Why This CMap Decision Strategy Matters

• Subset PDFs often ship ToUnicode tables referencing original GIDs that become invalid after font subsetting; the remapped CMap corrects GID→Unicode relationships
• Lazy evaluation via 240-byte sampling avoids full-document parsing overhead
• Per-object caching provides O(1) lookup for subsequent pages using the same font

Key Source Files

File	Purpose
src/extractor/fonts.rs	CMapDecisionCache implementation and heuristic scoring integration
src/tounicode.rs	ToUnicode stream parsing and TrueType fallback CMap construction
src/text_utils.rs	score_text function for Unicode quality assessment
src/extractor/mod.rs	Orchestration wiring caching into the text-decoding pipeline

Summary

• CMap decisions in pdf-inspector are made once per font object using a sampling-based quality heuristic
• The 240-byte threshold balances statistical reliability against extraction performance
• A 5-point scoring margin prevents switching to remapped maps without substantial evidence
• The per-object cache eliminates redundant analysis across multi-page documents

Frequently Asked Questions

What triggers pdf-inspector to choose the remapped CMap over the primary ToUnicode stream?

The remapped CMap is selected when its accumulated text sample scores more than 5 points higher than the primary sample on the score_text heuristic. This margin ensures the TrueType-based reconstruction is substantially more reliable, not merely marginally better.

How much PDF content does pdf-inspector need to analyze before making a CMap decision?

The extractor requires 240 bytes of source data (SAMPLE_TARGET_BYTES) before finalizing a choice. This threshold, implemented in src/extractor/fonts.rs, provides sufficient statistical signal for Unicode quality assessment without requiring full document parsing.

Can pdf-inspector's CMap decision change after the initial 240-byte sample?

No. Once consider accumulates enough data and stores a CMapChoice in the cache, all subsequent calls to get_choice for that object number return the cached result immediately. This design guarantees consistent text decoding across all pages using the same font.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}