CID Font Support in pdf-inspector: How It Decodes Type0/Identity-H Fonts

pdf-inspector handles CID-based Type0 fonts with Identity-H encoding through a three-stage fallback strategy: ToUnicode CMap parsing, embedded TrueType cmap extraction, and final CID-to-Unicode passthrough when CIDs directly encode Unicode scalar values.

Extracting readable text from modern PDFs often fails when generators like Chromium or wkhtmltopdf strip the ToUnicode mapping from CID-keyed fonts. The firecrawl/pdf-inspector repository solves this with a robust decoder that recovers text even from stripped Identity-H fonts while falling back to OCR when necessary. This article examines the complete CID font support implementation across the Rust source code.

Understanding the CID Font Problem

PDFs using Type0 fonts with Identity-H encoding store characters as 2-byte Character IDs (CIDs) rather than direct Unicode code points. The /ToUnicode CMap normally translates CIDs to Unicode, but many PDF generators remove this mapping to reduce file size. Without it, standard extractors produce gibberish or empty output.

pdf-inspector addresses this gap through coordinated detection, fallback construction, and intelligent passthrough decoding.

Stage 1: Font Detection and CMap Collection

The extraction pipeline begins in src/extractor/fonts.rs with collect_cmaps_from_fonts, which scans every font dictionary on a page to identify Type0 fonts using Identity-H or Identity-V encoding.

// src/extractor/fonts.rs, lines 31-44 — simplified conceptual flow
fn collect_cmaps_from_fonts(page: &Page) -> HashMap<u32, CMapEntry> {
    for font in page.fonts.values() {
        if let Some(font_dict) = font.as_dict() {
            // Check for Type0 with Identity-H/Identity-V encoding
            if is_type0_with_identity_encoding(font_dict) {
                let descendant = get_descendant_font(font_dict);
                let font_file_obj = get_font_file2_obj_num(descendant);
                // Use font_file_obj as key for fallback CMap
            }
        }
    }
}

The get_font_file2_obj_num function locates the embedded font data through the FontDescriptor → FontFile2 or FontFile3 chain. This object number becomes the key for associating fallback CMaps with specific fonts.

Stage 2: ToUnicode Parsing with Fallback Construction

When build_cmap_entry_from_stream in src/tounicode.rs cannot parse a valid or complete ToUnicode CMap, it constructs a fallback CMap containing:

• The partially parsed primary map
• An embedded TrueType cmap if available
• A built-in generic cmap as last resort

// src/tounicode.rs, lines 32-45 — fallback CMap construction
fn build_cmap_entry_from_stream(stream: Option<&Stream>) -> CMapEntry {
    match parse_tounicode_stream(stream) {
        Ok(map) if map.is_complete() => CMapEntry::primary(map),
        Ok(partial) => CMapEntry::with_fallback(partial, extract_truetype_cmap()),
        Err(_) => CMapEntry::fallback_only(built_in_cmap()),
    }
}

Stage 3: CID-as-Unicode Passthrough Fallback

The critical innovation for Identity-H support appears in src/tounicode.rs (lines 38-54). When no valid mapping exists, pdf-inspector analyzes the CID font's /W (widths) array:

• If CID values predominantly match Unicode code points (≥ 0x41 for ASCII, typical Unicode ranges)
• The system creates an empty ToUnicodeCMap with cid_passthrough = true

// src/tounicode.rs, lines 38-54 — CID passthrough detection
fn create_cid_passthrough_cmap(font_obj: u32, widths: &[u16]) -> ToUnicodeCMap {
    let mut cmap = ToUnicodeCMap::new();
    cmap.code_byte_length = 2;      // Identity-H always uses 2-byte CIDs
    cmap.cid_passthrough = true;    // Enable direct CID → Unicode casting
    
    // Analysis of widths array confirms Unicode-like CID distribution
    if appears_to_encode_unicode(widths) {
        cmap.register_for_font(font_obj);
        cmap
    } else {
        mark_for_ocr(font_obj)
    }
}

This heuristic works because many generators (notably Chromium's PDF engine) use Unicode scalar values directly as CIDs when constructing Identity-H fonts.

Stage 4: Runtime Byte Stream Decoding

The ToUnicodeCMap::decode_bytes method implements the actual decoding logic in src/tounicode.rs (lines 78-90). For 2-byte CID streams, it first attempts normal map lookup, then falls back to direct CID-to-Unicode conversion:

// src/tounicode.rs, lines 78-90 — the passthrough code path
impl ToUnicodeCMap {
    pub fn decode_bytes(&self, bytes: &[u8]) -> Option<String> {
        let mut result = String::new();
        for chunk in bytes.chunks(self.code_byte_length) {
            let cid = u16::from_be_bytes([chunk[0], chunk[1]]);
            
            if let Some(unicode) = self.map.get(&cid) {
                result.push(*unicode);
            } else if self.cid_passthrough {
                // Direct CID → Unicode passthrough for Identity-H fonts
                if let Some(ch) = char::from_u32(cid as u32) {
                    result.push(ch);
                }
            }
        }
        Some(result)
    }
}

The critical else branch enables text recovery when the ToUnicode map is missing but CIDs encode valid Unicode scalars.

Stage 5: OCR Fallback Detection

When passthrough fails or CIDs do not correspond to Unicode, src/detector.rs (around line 543) flags the page for OCR processing:

// src/detector.rs — Identity-H detection for OCR fallback
fn has_identity_h_without_tounicode(page: &Page) -> bool {
    page.fonts.values().any(|font| {
        let is_identity = font.encoding == "Identity-H" || font.encoding == "Identity-V";
        let undecodable = !font.has_valid_tounicode && !font.cid_passthrough_applied;
        is_identity && undecodable
    })
}

The detection occurs in should_use_ocr_for_page, which combines this check with other undecodability signals.

Practical Usage

pdf-inspector applies this entire pipeline automatically through its high-level API:

// Automatic CID font handling
let md = pdf_inspector::process_pdf_with_options("input.pdf", Default::default())?;
println!("{}", md);  // Text from Identity-H fonts correctly decoded

To inspect the internal CMap decisions for debugging:

use pdf_inspector::tounicode::ToUnicodeCMap;

// Manual passthrough CMap construction
let mut cmap = ToUnicodeCMap::new();
cmap.code_byte_length = 2;
cmap.cid_passthrough = true;

// Verify decoding behavior
let decoded = cmap.decode_bytes(&[0x00, 0x41]); // CID 65 → 'A'
assert_eq!(decoded, Some("A".to_string()));

Summary

• Font scanning in src/extractor/fonts.rs identifies Type0/Identity-H fonts and locates embedded font objects via get_font_file2_obj_num
• Multi-stage fallback in src/tounicode.rs attempts ToUnicode parsing, TrueType cmap extraction, then CID-passthrough construction
• Heuristic passthrough enables Unicode recovery when generators use CIDs as direct Unicode values, controlled by cid_passthrough flag
• Runtime decoding in ToUnicodeCMap::decode_bytes applies the passthrough in the 2-byte CID loop's fallback branch
• OCR detection in src/detector.rs marks pages as undecodable when all strategies fail, ensuring no silent data loss

Frequently Asked Questions

What is an Identity-H font in PDF?

An Identity-H font is a Type0 composite font where the encoding name /Identity-H indicates that CIDs map directly to horizontal glyph indices without additional CMap transformation. This encoding commonly appears in PDFs generated by web browsers. While efficient for rendering, it often lacks the /ToUnicode CMap needed for text extraction, which pdf-inspector addresses through heuristic passthrough decoding.

Why does pdf-inspector use CID passthrough instead of always using OCR?

CID passthrough preserves vector text accuracy at native resolution, whereas OCR introduces recognition errors, layout artifacts, and slower processing. The passthrough strategy succeeds for the majority of modern PDFs where CIDs encode Unicode directly. OCR serves only as a controlled fallback when the heuristic fails, maintaining extraction quality while maximizing coverage.

How does pdf-inspector detect when passthrough will fail?

pdf-inspector analyzes the CID font's /W (widths) array during create_cid_passthrough_cmap. If CID values fall outside expected Unicode ranges or show non-text distributions, the font is flagged as undecodable. The detector in src/detector.rs then combines this signal with other font properties to make page-level OCR decisions.

Can I force OCR for all Identity-H fonts?

The current API does not expose a direct "force OCR" flag for specific encodings. However, you can preprocess pages by inspecting CMapEntry construction results or by modifying the detector logic in a fork. The architecture separates detection (src/detector.rs) from extraction, making such customizations straightforward.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}