# How pdf-inspector Handles PDF Encodings with CMap Parsing: A Deep Dive into ToUnicode Extraction

> Discover how pdf-inspector expertly parses PDF encodings using CMap and extracts ToUnicode mappings for accurate text retrieval, even with malformed data.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-09-01

---

**pdf-inspector extracts readable text from PDFs by parsing the ToUnicode CMap that maps font-specific character IDs (CIDs) to Unicode strings, with robust fallback mechanisms for malformed or missing mappings.**

PDF documents encode text using font-specific character identifiers that bear no direct relationship to Unicode. The **ToUnicode CMap** bridges this gap, translating raw CID byte sequences into human-readable strings. Understanding how pdf-inspector processes these structures reveals the mechanics behind reliable PDF text extraction.

## What Is a ToUnicode CMap in PDF Processing?

A **CMap (Character Map)** is a PDF data structure that defines how character codes from a content stream map to character IDs, and ultimately to Unicode values. The **ToUnicode CMap** specifically provides the reverse mapping: from CIDs used by a font to UTF-16BE-encoded Unicode strings.

Without this mapping, extracting meaningful text from a PDF becomes impossible—you would only retrieve arbitrary byte values with no semantic meaning. pdf-inspector implements a complete CMap parsing pipeline in Rust to solve this problem.

## The Four-Stage CMap Parsing Pipeline

pdf-inspector processes ToUnicode CMaps through a strictly defined pipeline implemented in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs). Each stage transforms the raw PDF data into a usable lookup structure.

### Stage 1: Load and Parse the Raw CMap Stream

The `ToUnicodeCMap::parse` function reads the decompressed ToUnicode stream and builds internal data structures:

- Extracts **codespace ranges** that define valid byte sequences
- Parses `bfchar` entries for direct CID-to-Unicode mappings
- Parses `bfrange` entries for range-based mappings
- Populates the `char_map` hash map and `ranges` vector

```rust
// Reference implementation location: src/tounicode.rs, lines 103-210

```

The parser handles the PostScript-inspired syntax of CMap files, converting hexadecimal character codes into Rust-native byte representations.

### Stage 2: Determine Source-Byte Length

After parsing, pdf-inspector infers the **byte width** (`code_byte_length`) for the encoding:

- Derives from explicit codespace declarations when present
- Falls back to the longest hexadecimal entry found in the CMap
- Results in either 1-byte or 2-byte character IDs

This determination occurs at [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), lines 71-92. The byte width is critical: misinterpreting a 2-byte encoding as 1-byte produces gibberish output.

### Stage 3: Resolve CIDs to Unicode

The `ToUnicodeCMap::lookup` method performs the actual translation:

```rust
// Pseudocode based on src/tounicode.rs, lines 398-430
fn lookup(&self, cid: &[u8]) -> Option<String> {
    // First: direct hash map lookup
    if let Some(unicode) = self.char_map.get(cid) {
        return Some(unicode.clone());
    }
    // Second: binary search through sorted ranges
    self.ranges.binary_search_by(...)
}

```

The **binary search** through `ranges` ensures O(log n) performance even for CMaps defining thousands of contiguous ranges.

### Stage 4: Decode PDF Content Streams

The `decode_cids` function applies the CMap to raw content stream bytes:

```rust
// From src/tounicode.rs, lines 50-71
pub fn decode_cids(&self, bytes: &[u8]) -> String {
    // Walk bytes using determined code_byte_length
    // Fall back to Latin-1 for unmapped single-byte codes
    // Optional cid-passthrough for 2-byte codes (Identity-H fonts)
    // Return empty string if >50% codes unmapped
}

```

The **50% unmapped threshold** serves as a quality gate. When too many characters fail to resolve, pdf-inspector signals upstream code to attempt alternative decoding strategies rather than emit garbage.

## Fallback and Repair Strategies for Malformed CMaps

Real-world PDFs frequently contain broken, missing, or subset-corrupted ToUnicode CMaps. pdf-inspector implements three complementary recovery mechanisms.

### Fallback CMaps from TrueType Font Tables

When parsing fails entirely, pdf-inspector constructs alternative mappings:

| Function | Purpose |
|----------|---------|
| `build_fallback_cmap_for_type0` | Composite (Type 0) fonts with descendant CIDFonts |
| `build_fallback_cmap_for_simple` | Simple fonts with embedded TrueType data |
| `build_cmap_from_truetype` | Extracts `cmap`, `post` tables for glyph-to-Unicode mapping |
| `build_cmap_from_glyph_names` | Uses PostScript glyph name conventions |

These functions at [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), lines 69-95, leverage the font file itself rather than the PDF's ToUnicode stream.

### Builtin Binary CMaps (BCMaps)

pdf-inspector ships pre-compiled **Adobe-*-UCS2.bcmap** files in the `external/bcmaps/` directory. The `load_builtin_cmap_by_name` function (lines 1200-1209) merges these with parsed CMaps when a `usecmap` directive references them.

This provides authoritative mappings for standard Adobe character collections without parsing overhead.

### Subset-Font Remapping Repair

Subsetted fonts renumber glyph IDs to include only used characters, often breaking ToUnicode mappings. pdf-inspector detects and repairs this through `try_remap_subset_cmap`:

1. **Detection**: Identifies Identity-H/V fonts where minimum CID > 2 (indicating subsetting)
2. **CID→GID map repair**: Uses embedded `CIDToGIDMap` to reconstruct proper ordering
3. **Sequential remap**: Falls back to `remap_to_sequential` to assign CIDs 1, 2, 3...

Reference: [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), lines 86-106.

## Integration with the Text Extraction Pipeline

CMap handling connects to the broader extraction flow through three key components:

**`build_cmap_entry_from_stream`** orchestrates the entire process:
- Invokes the parser
- Triggers fallback construction on failure
- Attempts subset repair when indicated
- Returns a `CMapEntry` containing primary, remapped, and fallback variants

**[`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs)** integrates CMap resolution with font dictionary processing, selecting the appropriate variant based on decoding success.

**[`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs)** walks PDF content operators (`Tj`, `TJ`) and supplies raw byte streams to `cmap.decode_cids`, appending results to Markdown output.

## Practical Code Examples

### Loading a ToUnicode CMap from PDF Data

```rust
let cmap_entry = tounicode::build_cmap_entry_from_stream(
    &stream_data,          // decompressed stream bytes
    &font_dict,            // lopdf::Dictionary for the font
    &doc,                  // lopdf::Document reference
    obj_num,               // object number for diagnostics
);

// cmap_entry.primary: parsed or fallback CMap
// cmap_entry.remapped: subset-repaired variant if applicable
// cmap_entry.fallback: TrueType-derived backup if primary failed

```

### Decoding Content Stream Bytes

```rust
let cmap = cmap_entry.primary;        // Select best available CMap
let utf8 = cmap.decode_cids(&bytes);  // Convert CIDs to Unicode

// Empty result triggers alternative selection:
if utf8.is_empty() && cmap_entry.remapped.is_some() {
    let utf8 = cmap_entry.remapped.unwrap().decode_cids(&bytes);
}

```

### Manual Subset Repair Trigger

```rust
let (repaired_cmap, did_repair) = tounicode::try_remap_subset_cmap(
    original_cmap,
    &font_dict,
    &doc,
    obj_num,
);

if did_repair {
    // Use repaired_cmap instead
}

```

## Key Source Files

| File | Responsibility |
|------|---------------|
| [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) | Core CMap parsing, lookup, decoding, and all repair logic |
| [`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs) | Font dictionary processing and CMap selection |
| [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs) | Content stream traversal and text operator handling |
| `external/bcmaps/` | Pre-compiled Adobe binary CMap resources |

## Summary

- **ToUnicode CMap parsing** in pdf-inspector follows a four-stage pipeline: stream parsing → byte length determination → CID-to-Unicode lookup → content stream decoding
- **`decode_cids`** implements quality gating via the 50% unmapped threshold, enabling graceful degradation
- **Three fallback layers** protect against real-world PDF damage: TrueType table extraction → builtin binary CMaps → subset-font repair heuristics
- **Performance-conscious design** uses hash maps for direct lookups and binary search for range queries

## Frequently Asked Questions

### How does pdf-inspector handle PDFs without ToUnicode CMaps?

pdf-inspector constructs fallback CMaps by extracting glyph-to-Unicode mappings directly from embedded TrueType font tables (`cmap`, `post`) or inferring from PostScript glyph naming conventions. The functions `build_cmap_from_truetype` and `build_cmap_from_glyph_names` implement this recovery at [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), lines 69-95.

### What causes the "empty string" return from `decode_cids`?

When more than 50% of character codes fail to map to Unicode, `decode_cids` returns an empty string to signal decoding failure. This threshold prevents output of partial gibberish and triggers upstream code to select an alternative CMap variant (remapped or fallback) or extraction method.

### How does pdf-inspector detect and repair subsetted fonts?

Subsetted fonts are detected by checking for Identity-H/V encoding with minimum CID greater than 2. The `try_remap_subset_cmap` function first attempts repair using an embedded `CIDToGIDMap`, then falls back to sequential remapping via `remap_to_sequential` to restore linear CID ordering.

### Where are the builtin binary CMaps stored and loaded?

Pre-compiled Adobe CMaps reside in `external/bcmaps/` as binary `.bcmap` files. The `load_builtin_cmap_by_name` function at [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), lines 1200-1209, locates and merges these when a CMap stream contains a `usecmap` directive referencing a standard Adobe character collection name.