How pdf-inspector Resolves CMaps and Font Information: A Deep Dive into PDF Text Extraction

pdf-inspector converts CID values in PDF content streams into Unicode characters through a multi-layered CMap resolution pipeline that prioritizes embedded ToUnicode data, repairs subset-font mismatches, and falls back to TrueType tables, glyph names, and built-in binary CMaps.

Accurate text extraction from PDFs depends entirely on resolving how character identifiers (CIDs) map to actual Unicode characters. The firecrawl/pdf-inspector project implements a robust, layered approach in Rust that handles the messy reality of PDF fonts—missing CMaps, subset font renumbering, and inconsistent encoding. This article breaks down exactly how the library resolves CMap and font information, with direct references to the source code implementation.

Core Architecture of CMap Resolution in pdf-inspector

The font resolution pipeline follows a strict priority order. Each layer attempts to build a complete ToUnicodeCMap, and the system automatically promotes the most reliable source:

Embedded ToUnicode → Subset repair → TrueType cmap → Glyph names → Built-in CMap

This design ensures that even malformed or minimally-structured PDFs yield readable text. The orchestration happens primarily in src/extractor/fonts.rs, while the low-level parsing logic resides in src/tounicode.rs.

Parsing Embedded ToUnicode CMap Streams

The first and preferred source is the font's own /ToUnicode entry, which contains a CMap stream defining CID-to-Unicode mappings.

ToUnicodeCMap::parse Implementation

In src/tounicode.rs, the ToUnicodeCMap::parse function handles the full CMap syntax:

• begincodespacerange: Defines valid byte lengths (1-byte or 2-byte CIDs)
• beginbfchar: Direct CID→Unicode mappings
• beginbfrange: Range-based mappings for efficiency
• usecmap: Merges in referenced built-in CMaps

The parser builds two internal structures:

• char_map: BTreeMap<u16, String> — direct lookups
• ranges: Vec<RangeMapping> — range-based lookups with linear search fallback

// Minimal example of using the ToUnicode parser directly
use pdf_inspector::tounicode::ToUnicodeCMap;

let raw_cmap: &[u8] = /* bytes from a PDF's /ToUnicode stream */;
if let Some(cmap) = ToUnicodeCMap::parse(raw_cmap) {
    // Convert a CID to Unicode
    let unicode = cmap.lookup(0x12).unwrap_or_default();
    println!("CID 0x12 → {}", unicode);
}

Detecting and Repairing Subset Font Mismatches

PDF creators frequently embed subset fonts where glyph IDs are renumbered to save space. This breaks CMap validity when the ToUnicode stream still references original GIDs.

try_remap_subset_cmap in src/tounicode.rs

The try_remap_subset_cmap function detects Identity-H/V encoding with mismatched CID ranges and applies two repair strategies:

1. CIDToGIDMap reversal: If the font contains a CIDToGIDMap, the code rebuilds the CMap by reversing this mapping to obtain correct CID→Unicode associations.

2. Sequential remapping: When no CIDToGIDMap exists, remap_to_sequential reassigns CIDs to consecutive values, assuming the subset was constructed linearly.

Both paths log decisions via debug! and warn! macros for traceability.

Fallback to TrueType cmap Tables

When embedded ToUnicode data is absent or sparse, pdf-inspector extracts the font's TrueType cmap table as an alternative Unicode source.

CID Fonts: build_cmap_from_truetype

For Identity-H encoded fonts (common in Asian-language PDFs), the implementation:

• Extracts the Unicode→GID subtable
• Reverses the mapping to create GID→Unicode (effectively CID→Unicode)

Simple Fonts: build_simple_cmap_from_truetype

For single-byte fonts, build_simple_cmap_from_truetype attempts multiple platform-specific subtables in priority order:

1. MacRoman (platform ID 1, encoding ID 0)
2. Windows Symbol (platform ID 3, encoding ID 0)
3. Windows Unicode BMP (platform ID 3, encoding ID 1)

Each byte code is mapped through the selected subtable to obtain a glyph ID, then resolved to Unicode.

Glyph Name Resolution via post Table

If TrueType tables fail, pdf-inspector falls back to glyph names from the font's post table.

build_cmap_from_glyph_names Implementation

In src/tounicode.rs, this function:

• Iterates named glyphs in the post table
• Converts each name to Unicode using glyph_to_char from src/glyph_names.rs

The glyph_to_char function implements the Adobe Glyph List (AGL), translating standard glyph names like uni0041 or A to their corresponding codepoints.

Built-in Binary CMaps (bcmaps)

As a final resort, pdf-inspector loads pre-compiled binary CMaps from external/bcmaps/.

Binary CMap Loading Functions

• read_builtin_cmap_file: Locates and reads .bcmap files
• parse_binary_cmap: Parses the compact binary format
• load_builtin_cmap_by_name: Loads standard names like Adobe-Identity-UCS2.bcmap

These binary CMaps provide baseline Unicode coverage for fonts that lack any embedded mapping information.

Final CMap Selection Logic

The build_cmap_entry_from_stream function in src/tounicode.rs implements intelligent selection between candidate CMaps:

Condition	Action
Primary CMap has < 10 entries	Prefer fallback CMap
TrueType fallback has more entries	Promote fallback to primary
Remapped version available	Consider if subset repair succeeded

This heuristic ensures that completeness—not just source priority—determines which mapping table drives text extraction.

Decoding Content Stream Bytes with Resolved CMaps

With the final ToUnicodeCMap selected, decode_cids performs the actual byte-to-text conversion:

• Direct lookup: Fast path for char_map entries
• Range search: Binary search through ranges for range-based codes
• Unmapped handling: Logs warnings; optionally applies Latin-1 fallback for single-byte fonts or CID passthrough (treating the CID as raw Unicode)

Using pdf-inspector for Text Extraction

# Basic extraction (text → markdown)

pdf2md my-document.pdf

# Request structured JSON output (includes per-font CMap info)

pdf2md --json my-document.pdf

The JSON output exposes the resolved CMap metadata, enabling debugging of font resolution decisions.

Summary

• Primary resolution: ToUnicodeCMap::parse handles embedded /ToUnicode streams with full CMap syntax support
• Subset repair: try_remap_subset_cmap fixes renumbered GIDs using CIDToGIDMap or sequential remapping
• TrueType fallback: build_cmap_from_truetype and build_simple_cmap_from_truetype extract cmap tables with platform-specific subtable priority
• Glyph name fallback: build_cmap_from_glyph_names uses Adobe Glyph List translations from the post table
• Last resort: load_builtin_cmap_by_name loads binary CMaps from external/bcmaps/
• Selection logic: build_cmap_entry_from_stream promotes the most complete mapping based on entry count heuristics
• Decoding: decode_cids applies the final CMap with range search and unmapped-code handling

Frequently Asked Questions

What is a CMap in PDF font terms?

A CMap (Character Map) defines how character identifiers (CIDs) in PDF content streams translate to Unicode characters. PDFs use CMaps because font glyphs are indexed by arbitrary IDs rather than standard Unicode codepoints—especially for CJK fonts and subsetted fonts. Without CMap resolution, extracted text appears as garbage or CID placeholders.

Why does pdf-inspector need multiple fallback strategies for CMap resolution?

PDF generation tools vary wildly in conformance. Many commercial PDFs contain incomplete or missing ToUnicode streams, especially for legacy documents or scanned-image PDFs with invisible text layers. The layered fallback—from embedded CMap through TrueType tables to built-in binaries—ensures maximum text recoverability across the full spectrum of real-world PDF quality.

How can I debug CMap resolution in pdf-inspector?

Run with RUST_LOG=debug to see detailed logging from tounicode.rs. The --json flag on pdf2md outputs per-font CMap metadata including which source was selected and entry counts. Check src/extractor/fonts.rs for the FontCMaps structure that stores resolution results.

What is the difference between Identity-H and simple font encoding?

Identity-H is a CID-keyed encoding used for CJK and other multi-byte fonts, where CIDs directly index glyphs. Simple encoding (WinAnsi, MacRoman, etc.) uses single-byte codes mapped through an encoding dictionary. pdf-inspector handles both through distinct codepaths: build_cmap_from_truetype for Identity-H reversal, and build_simple_cmap_from_truetype for single-byte subtable extraction.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}