How ToUnicode CMap Parsing Decodes CID-Encoded Fonts (Type0 and Identity-H) in pdf-inspector

ToUnicode CMap parsing converts CID-encoded fonts to Unicode by reading the ToUnicode stream, determining 1-byte or 2-byte code widths, building fallback mappings from TrueType tables, and handling CID-to-GID remapping for subsetted fonts.

pdf-inspector is an open-source Rust library for extracting text from PDF documents. When working with CID-encoded fonts—specifically Type0 fonts using Identity-H or Identity-V encodings—the library must interpret character IDs (CIDs) rather than simple byte sequences. This article explains the complete decoding pipeline implemented in the source code.

Overview of CID Font Decoding

CID fonts represent glyphs as Character IDs instead of direct character codes. The ToUnicode CMap attached to a PDF font provides the mapping from these CIDs to Unicode code points. The pdf-inspector implementation follows a six-step process to reliably extract text even from poorly-encoded or subsetted documents.

Step 1: Parse the Raw ToUnicode Stream

The entry point build_cmap_entry_from_stream in src/tounicode.rs reads the decompressed ToUnicode stream and constructs a ToUnicodeCMap object.

// src/tounicode.rs, lines 32-44
pub fn build_cmap_entry_from_stream(stream: &[u8]) -> Option<ToUnicodeCMap> {
    ToUnicodeCMap::parse(stream).ok()
}

The parse method extracts three critical sections from the CMap:

• begincodespacerange — defines valid code ranges
• bfchar — single character mappings
• bfrange — range-based character mappings

The parser also records source hex lengths to determine the code byte width in subsequent steps.

Step 2: Determine the Code Byte Width

CID fonts require 2-byte big-endian CIDs, while simple fonts may use single-byte codes. After parsing, the code calculates code_byte_length by analyzing the ranges found in the CMap.

// src/tounicode.rs, lines 71-92
impl ToUnicodeCMap {
    fn parse(data: &[u8]) -> Result<Self, CMapError> {
        let mut cmap = ToUnicodeCMap::default();
        // ... parsing logic ...
        
        // Determine byte width from maximum code value
        cmap.code_byte_length = if max_code <= 0xFF { 1 } else { 2 };
        Ok(cmap)
    }
}

This distinction is essential because Type0 fonts always emit 2-byte CIDs, and misinterpreting the width corrupts the text extraction.

Step 3: Build Fallback Mappings

When the ToUnicode CMap is missing or incomplete, pdf-inspector constructs fallback mappings from the embedded font data:

Fallback Type	Function	Source Location
TrueType cmap table	build_cmap_from_truetype	src/tounicode.rs:67-71
Simple font cmap	build_simple_cmap_from_truetype	Same file

These functions extract the cmap table from TrueType fonts, reverse the GID→Unicode mapping, and synthesize a functional ToUnicodeCMap.

Step 4: Handle CID-to-GID Mismatches in Subsetted Fonts

PDF generators often subset fonts by renumbering GIDs while preserving the original ToUnicode CMap. This creates a mismatch between CIDs in the content stream and entries in the CMap.

The try_remap_subset_cmap function detects this condition specifically for Identity-H/V fonts:

// src/tounicode.rs, lines 80-86
pub fn try_remap_subset_cmap(
    cmap: ToUnicodeCMap,
    font_dict: &Dictionary,
    doc: &Document,
    obj_number: u32,
) -> (ToUnicodeCMap, Option<ToUnicodeCMap>) {
    // Check if font encoding is Identity-H or Identity-V
    // and if CID range suggests subsetting
    if is_subsetted(&cmap) {
        // Attempt remapping using CIDToGIDMap or sequential GIDs
        return remap_to_sequential(cmap);
    }
    (cmap, None)
}

The remapping strategy uses either:

• CIDToGIDMap — if present in the font descriptor, directly maps CIDs to correct GIDs
• Sequential remapping — assumes GIDs were renumbered sequentially starting from 0

Step 5: Decode CID Bytes to Unicode

The decode_cids function processes raw bytes from PDF content streams according to the determined code_byte_length:

// src/tounicode.rs, lines 50-66
impl ToUnicodeCMap {
    pub fn decode_cids(&self, bytes: &[u8]) -> String {
        let mut result = String::new();
        let mut i = 0;
        
        while i < bytes.len() {
            let (cid, consumed) = if self.code_byte_length == 2 {
                // Big-endian 2-byte CID
                let cid = ((bytes[i] as u16) << 8) | bytes[i + 1] as u16;
                (cid, 2)
            } else {
                (bytes[i] as u16, 1)
            };
            
            match self.lookup(cid) {
                Some(unicode) => result.push_str(&unicode),
                None if self.cid_passthrough => {
                    // Last resort: treat CID as Unicode code point
                    if let Some(c) = char::from_u32(cid as u32) {
                        result.push(c);
                    }
                }
                None => { /* unmapped */ }
            }
            i += consumed;
        }
        result
    }
}

For Identity-H fonts, the cid_passthrough flag enables direct CID-to-Unicode conversion when the CMap lookup fails. This works because Identity-H encoding uses Unicode values as CIDs.

Step 6: Quality Check and Final Fallback

If more than half of the CIDs remain unmapped, decode_cids returns an empty string (lines 103-111), signaling the caller to try alternative strategies such as glyph name extraction.

Why This Works for Type0 and Identity-H/V

• Type0 fonts are inherently CID-based; the encoding determines how CIDs map to glyphs
• Identity-H/V indicates Unicode values were used directly as CIDs, making passthrough decoding viable
• The 2-byte code width assumption aligns with the big-endian CID convention in PDF specifications

Summary

• build_cmap_entry_from_stream initiates parsing from raw PDF streams
• code_byte_length detection distinguishes 1-byte simple fonts from 2-byte CID fonts
• TrueType cmap tables provide fallback mappings when ToUnicode is absent
• try_remap_subset_cmap corrects GID renumbering in subsetted Identity-H/V fonts
• decode_cids performs the final CID→Unicode conversion with passthrough support
• Unmapped code thresholds trigger alternative extraction strategies

Frequently Asked Questions

What is a ToUnicode CMap in PDF files?

A ToUnicode CMap is a mapping embedded in PDF fonts that converts character identifiers (CIDs or character codes) to Unicode code points. Without this mapping, text extraction tools cannot determine what Unicode characters correspond to the glyphs displayed in the PDF. According to the pdf-inspector source code, these CMaps contain begincodespacerange, bfchar, and bfrange sections that define the decoding rules.

Why do Type0 fonts require special handling compared to simple fonts?

Type0 fonts are Composite fonts that use CIDs instead of direct character codes. The PDF content stream contains 2-byte big-endian CID values rather than ASCII or single-byte encodings. The pdf-inspector implementation detects this through code_byte_length == 2 and processes the bytes accordingly in decode_cids. Simple fonts typically use 1-byte codes that map directly to glyph indices.

What happens when a PDF has no ToUnicode CMap?

When the ToUnicode CMap is missing, pdf-inspector executes fallback logic in build_cmap_from_truetype or build_simple_cmap_from_truetype. These functions extract the cmap table from embedded TrueType font data and construct a reverse mapping from GID to Unicode. For Identity-H/V fonts, the code additionally enables cid_passthrough mode, treating CIDs directly as Unicode code points when other methods fail.

How does pdf-inspector handle font subsetting?

Font subsetting renumbers GIDs while often preserving the original ToUnicode CMap, causing CID-to-GID misalignment. The try_remap_subset_cmap function detects this by checking if the minimum CID in the CMap exceeds expected values for Identity-H/V encodings. It then applies either a CIDToGIDMap from the font descriptor or sequential remapping to realign the CMap entries with the actual subsetted GIDs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}