How pdf-inspector Handles PDF Encodings with CMap Parsing: A Deep Dive into ToUnicode Extraction

pdf-inspector extracts readable text from PDFs by parsing the ToUnicode CMap that maps font-specific character IDs (CIDs) to Unicode strings, with robust fallback mechanisms for malformed or missing mappings.

PDF documents encode text using font-specific character identifiers that bear no direct relationship to Unicode. The ToUnicode CMap bridges this gap, translating raw CID byte sequences into human-readable strings. Understanding how pdf-inspector processes these structures reveals the mechanics behind reliable PDF text extraction.

What Is a ToUnicode CMap in PDF Processing?

A CMap (Character Map) is a PDF data structure that defines how character codes from a content stream map to character IDs, and ultimately to Unicode values. The ToUnicode CMap specifically provides the reverse mapping: from CIDs used by a font to UTF-16BE-encoded Unicode strings.

Without this mapping, extracting meaningful text from a PDF becomes impossible—you would only retrieve arbitrary byte values with no semantic meaning. pdf-inspector implements a complete CMap parsing pipeline in Rust to solve this problem.

The Four-Stage CMap Parsing Pipeline

pdf-inspector processes ToUnicode CMaps through a strictly defined pipeline implemented in src/tounicode.rs. Each stage transforms the raw PDF data into a usable lookup structure.

Stage 1: Load and Parse the Raw CMap Stream

The ToUnicodeCMap::parse function reads the decompressed ToUnicode stream and builds internal data structures:

• Extracts codespace ranges that define valid byte sequences
• Parses bfchar entries for direct CID-to-Unicode mappings
• Parses bfrange entries for range-based mappings
• Populates the char_map hash map and ranges vector

// Reference implementation location: src/tounicode.rs, lines 103-210

The parser handles the PostScript-inspired syntax of CMap files, converting hexadecimal character codes into Rust-native byte representations.

Stage 2: Determine Source-Byte Length

After parsing, pdf-inspector infers the byte width (code_byte_length) for the encoding:

• Derives from explicit codespace declarations when present
• Falls back to the longest hexadecimal entry found in the CMap
• Results in either 1-byte or 2-byte character IDs

This determination occurs at src/tounicode.rs, lines 71-92. The byte width is critical: misinterpreting a 2-byte encoding as 1-byte produces gibberish output.

Stage 3: Resolve CIDs to Unicode

The ToUnicodeCMap::lookup method performs the actual translation:

// Pseudocode based on src/tounicode.rs, lines 398-430
fn lookup(&self, cid: &[u8]) -> Option<String> {
    // First: direct hash map lookup
    if let Some(unicode) = self.char_map.get(cid) {
        return Some(unicode.clone());
    }
    // Second: binary search through sorted ranges
    self.ranges.binary_search_by(...)
}

The binary search through ranges ensures O(log n) performance even for CMaps defining thousands of contiguous ranges.

Stage 4: Decode PDF Content Streams

The decode_cids function applies the CMap to raw content stream bytes:

// From src/tounicode.rs, lines 50-71
pub fn decode_cids(&self, bytes: &[u8]) -> String {
    // Walk bytes using determined code_byte_length
    // Fall back to Latin-1 for unmapped single-byte codes
    // Optional cid-passthrough for 2-byte codes (Identity-H fonts)
    // Return empty string if >50% codes unmapped
}

The 50% unmapped threshold serves as a quality gate. When too many characters fail to resolve, pdf-inspector signals upstream code to attempt alternative decoding strategies rather than emit garbage.

Fallback and Repair Strategies for Malformed CMaps

Real-world PDFs frequently contain broken, missing, or subset-corrupted ToUnicode CMaps. pdf-inspector implements three complementary recovery mechanisms.

Fallback CMaps from TrueType Font Tables

When parsing fails entirely, pdf-inspector constructs alternative mappings:

Function	Purpose
build_fallback_cmap_for_type0	Composite (Type 0) fonts with descendant CIDFonts
build_fallback_cmap_for_simple	Simple fonts with embedded TrueType data
build_cmap_from_truetype	Extracts cmap, post tables for glyph-to-Unicode mapping
build_cmap_from_glyph_names	Uses PostScript glyph name conventions

These functions at src/tounicode.rs, lines 69-95, leverage the font file itself rather than the PDF's ToUnicode stream.

Builtin Binary CMaps (BCMaps)

pdf-inspector ships pre-compiled Adobe-*-UCS2.bcmap files in the external/bcmaps/ directory. The load_builtin_cmap_by_name function (lines 1200-1209) merges these with parsed CMaps when a usecmap directive references them.

This provides authoritative mappings for standard Adobe character collections without parsing overhead.

Subset-Font Remapping Repair

Subsetted fonts renumber glyph IDs to include only used characters, often breaking ToUnicode mappings. pdf-inspector detects and repairs this through try_remap_subset_cmap:

1. Detection: Identifies Identity-H/V fonts where minimum CID > 2 (indicating subsetting)
2. CID→GID map repair: Uses embedded CIDToGIDMap to reconstruct proper ordering
3. Sequential remap: Falls back to remap_to_sequential to assign CIDs 1, 2, 3...

Reference: src/tounicode.rs, lines 86-106.

Integration with the Text Extraction Pipeline

CMap handling connects to the broader extraction flow through three key components:

build_cmap_entry_from_stream orchestrates the entire process:

• Invokes the parser
• Triggers fallback construction on failure
• Attempts subset repair when indicated
• Returns a CMapEntry containing primary, remapped, and fallback variants

src/extractor/fonts.rs integrates CMap resolution with font dictionary processing, selecting the appropriate variant based on decoding success.

src/extractor/content_stream.rs walks PDF content operators (Tj, TJ) and supplies raw byte streams to cmap.decode_cids, appending results to Markdown output.

Practical Code Examples

Loading a ToUnicode CMap from PDF Data

let cmap_entry = tounicode::build_cmap_entry_from_stream(
    &stream_data,          // decompressed stream bytes
    &font_dict,            // lopdf::Dictionary for the font
    &doc,                  // lopdf::Document reference
    obj_num,               // object number for diagnostics
);

// cmap_entry.primary: parsed or fallback CMap
// cmap_entry.remapped: subset-repaired variant if applicable
// cmap_entry.fallback: TrueType-derived backup if primary failed

Decoding Content Stream Bytes

let cmap = cmap_entry.primary;        // Select best available CMap
let utf8 = cmap.decode_cids(&bytes);  // Convert CIDs to Unicode

// Empty result triggers alternative selection:
if utf8.is_empty() && cmap_entry.remapped.is_some() {
    let utf8 = cmap_entry.remapped.unwrap().decode_cids(&bytes);
}

Manual Subset Repair Trigger

let (repaired_cmap, did_repair) = tounicode::try_remap_subset_cmap(
    original_cmap,
    &font_dict,
    &doc,
    obj_num,
);

if did_repair {
    // Use repaired_cmap instead
}

Key Source Files

File	Responsibility
src/tounicode.rs	Core CMap parsing, lookup, decoding, and all repair logic
src/extractor/fonts.rs	Font dictionary processing and CMap selection
src/extractor/content_stream.rs	Content stream traversal and text operator handling
external/bcmaps/	Pre-compiled Adobe binary CMap resources

Summary

• ToUnicode CMap parsing in pdf-inspector follows a four-stage pipeline: stream parsing → byte length determination → CID-to-Unicode lookup → content stream decoding
• decode_cids implements quality gating via the 50% unmapped threshold, enabling graceful degradation
• Three fallback layers protect against real-world PDF damage: TrueType table extraction → builtin binary CMaps → subset-font repair heuristics
• Performance-conscious design uses hash maps for direct lookups and binary search for range queries

Frequently Asked Questions

How does pdf-inspector handle PDFs without ToUnicode CMaps?

pdf-inspector constructs fallback CMaps by extracting glyph-to-Unicode mappings directly from embedded TrueType font tables (cmap, post) or inferring from PostScript glyph naming conventions. The functions build_cmap_from_truetype and build_cmap_from_glyph_names implement this recovery at src/tounicode.rs, lines 69-95.

What causes the "empty string" return from decode_cids?

When more than 50% of character codes fail to map to Unicode, decode_cids returns an empty string to signal decoding failure. This threshold prevents output of partial gibberish and triggers upstream code to select an alternative CMap variant (remapped or fallback) or extraction method.

How does pdf-inspector detect and repair subsetted fonts?

Subsetted fonts are detected by checking for Identity-H/V encoding with minimum CID greater than 2. The try_remap_subset_cmap function first attempts repair using an embedded CIDToGIDMap, then falls back to sequential remapping via remap_to_sequential to restore linear CID ordering.

Where are the builtin binary CMaps stored and loaded?

Pre-compiled Adobe CMaps reside in external/bcmaps/ as binary .bcmap files. The load_builtin_cmap_by_name function at src/tounicode.rs, lines 1200-1209, locates and merges these when a CMap stream contains a usecmap directive referencing a standard Adobe character collection name.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}