How pdf-inspector Handles CMap and ToUnicode Parsing for Font Decoding

pdf-inspector implements a four-tier fallback strategy in src/tounicode.rs that converts PDF character identifiers (CIDs) into Unicode strings by parsing /ToUnicode streams, binary CMaps, built-in encodings, TrueType tables, and glyph names.

Extracting readable text from PDFs requires decoding font-specific character identifiers into Unicode, a process governed by CMap and ToUnicode parsing. The pdf-inspector library, developed by Firecrawl as a Rust-based text extraction engine, solves this through a robust pipeline implemented primarily in src/tounicode.rs. This system handles everything from standard ASCII CMaps to compact binary formats and subset font remapping, ensuring accurate text extraction even from malformed or strictly encoded documents.

The Core Architecture

The font decoding system centers on two primary modules. The src/tounicode.rs file contains the low-level parsing logic for CMap structures, while src/extractor/fonts.rs orchestrates the FontCMaps structure that manages the layered fallback strategy. When processing a PDF page, the extractor builds a FontCMaps instance for each font object, which attempts to acquire the most complete Unicode mapping available through a prioritized sequence of sources.

The pipeline handles both ASCII-based CMap syntax and compact binary CMap formats (such as Adobe-Japan1), detecting the format automatically during the parse phase. For each text-showing operator encountered in the content stream, the system calls ToUnicodeCMap::decode or lookup_cid to transform raw byte sequences into valid Unicode strings.

The CMap Parsing Pipeline

Loading the ToUnicode Stream

For every font object, pdf-inspector first searches for a /ToUnicode entry in the font dictionary. When present, the raw stream bytes are passed to ToUnicodeCMap::parse (lines 103-130 in src/tounicode.rs), which interprets the PDF CMap syntax regardless of whether it uses ASCII or binary encoding. This function serves as the primary entry point for character mapping discovery.

Handling Binary CMap Formats

Many professional PDFs embed compact binary CMaps to reduce file size. The parse_binary_cmap function (lines 1244-1320) processes these streams through a BinaryCMapStream reader, extracting the code-byte length, character-to-Unicode ranges, and populating internal char_map and ranges tables. This allows the library to decode complex CJK (Chinese, Japanese, Korean) fonts that rely on predefined binary mappings.

Built-in CMap Fallbacks

When the /ToUnicode stream is missing or contains sparse entries, pdf-inspector falls back to pre-built CMaps shipped with the binary. The load_builtin_cmap_by_name function (lines 1870-1885) retrieves standardized maps such as Adobe-Korea1 or Adobe-Japan1 from embedded resources. These built-in maps undergo the same parsing pipeline as binary streams, ensuring consistent behavior across all CMap sources.

Fallback Strategies for Robust Decoding

If the primary ToUnicode mapping proves insufficient, pdf-inspector activates a sequence of tertiary extraction methods designed to handle edge cases in PDF generation.

TrueType cmap Table Extraction

When ToUnicode data is unavailable, the library probes embedded TrueType fonts via /FontDescriptor/FontFile2. The build_cmap_from_truetype function (lines 967-990) uses ttf_parser to read the font's internal cmap table, constructing a CID-to-Unicode map directly from glyph IDs. This approach recovers text from PDFs generated by applications that neglect to include explicit ToUnicode CMaps.

Adobe Glyph List Resolution

Some fonts expose only glyph names through the post table rather than explicit Unicode mappings. The build_cmap_from_glyph_names function (lines 1134-1150) maps these names to Unicode using the Adobe Glyph List (AGL), enabling text extraction from symbolic or decorative fonts that lack standard encoding tables.

Subset Font Remapping

PDF generators frequently subset fonts, causing the CID space to become non-sequential. The remap_to_sequential function (lines 548-566) detects these mismatches—particularly when a CIDToGIDMap is present—and rewrites the CMap so that CIDs 1 through N correctly map to their corresponding Unicode values. This correction is essential for documents using embedded subsets of larger typefaces.

Merging Multiple CMap Sources

Complex fonts may contain several CMap variants simultaneously. The merge_cmaps function (lines 1883-1898) combines primary, remapped, and fallback mappings into a unified structure, preferring the most complete mapping while preserving alternatives for error recovery. This ensures that even partially corrupt CMaps can contribute valid decoding information.

Text Decoding in Content Streams

During PDF content-stream processing in src/extractor/content_stream.rs, each text operator (Tj, TJ, Td, Tm) emits raw byte sequences. The extractor retrieves the appropriate FontCMaps entry for the current font, then invokes ToUnicodeCMap::decode (lines 450-460) or lookup_cid to convert these bytes into Unicode strings. This decoded text feeds directly into the markdown conversion pipeline, preserving the document's logical reading order and character set.

Practical Implementation Examples

High-Level PDF Text Extraction

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::ExtractionOptions;

let options = ExtractionOptions::default();
let markdown = process_pdf_with_options("sample.pdf", options).unwrap();
println!("{}", markdown);

This API automatically builds FontCMaps for each detected font and applies the full decoding pipeline during extraction.

Manual CMap Inspection

use pdf_inspector::extractor::fonts::FontCMaps;
use lopdf::Document;

let doc = Document::load("sample.pdf")?;
let font_dicts = /* collect Font objects from the page */;
let cmap_collection = FontCMaps::build(&doc, &font_dicts, None, false)?;

// Inspect the primary CMap of the first font
let entry = cmap_collection.by_obj_num.values().next().unwrap();
println!("Primary entries: {}", entry.primary.char_map.len());
println!("Remapped? {}", entry.remapped.is_some());

Raw CID Sequence Decoding

use pdf_inspector::tounicode::ToUnicodeCMap;

// Assuming a parsed CMap `cmap`
let raw_bytes = vec![0x00, 0x41]; // CID for "A" in a 2-byte CMap
let text = cmap.decode(&raw_bytes).unwrap();
assert_eq!(text, "A");

Summary

• pdf-inspector implements a four-tier fallback strategy: ToUnicode CMaps, built-in CMaps, TrueType tables, and glyph-name mapping.
• The core parsing logic resides in src/tounicode.rs, with ToUnicodeCMap::parse handling both ASCII and binary CMap formats.
• Binary CMap support enables proper decoding of CJK fonts through parse_binary_cmap and BinaryCMapStream.
• Subset font handling via remap_to_sequential corrects non-sequential CID spaces common in optimized PDFs.
• The architecture combines these sources through merge_cmaps to maximize text recovery from malformed or minimally encoded documents.

Frequently Asked Questions

What is a ToUnicode CMap in PDF processing?

A ToUnicode CMap is a PDF stream object that maps character identifiers (CIDs) used in content streams to standardized Unicode code points. According to the pdf-inspector source code, this mapping is essential because PDFs often use custom font encodings where the byte value 0x41 might not represent the Latin letter "A". The ToUnicodeCMap::parse function in src/tounicode.rs handles both ASCII and binary representations of these mappings.

How does pdf-inspector handle binary versus ASCII CMap streams?

The library automatically detects the format during parsing. ASCII CMaps are parsed directly by the ToUnicodeCMap::parse function, while binary CMaps are routed to parse_binary_cmap (lines 1244-1320), which uses a specialized BinaryCMapStream reader to extract code lengths and character ranges. Both formats ultimately populate the same internal char_map structures for consistent lookup performance.

What happens when a PDF font lacks a ToUnicode entry?

When the /ToUnicode stream is missing or sparse, pdf-inspector activates its fallback chain. First, it attempts to load built-in CMaps via load_builtin_cmap_by_name. If that fails, it probes the embedded TrueType font's cmap table using build_cmap_from_truetype. As a final resort, it uses build_cmap_from_glyph_names to map glyph names to Unicode via the Adobe Glyph List. The system only skips a font when all four methods fail to produce viable mappings.

How does the library handle subsetted fonts with non-sequential CIDs?

PDF generators often subset fonts by remapping glyph indices, causing CIDs to skip values or appear out of order. The remap_to_sequential function (lines 548-566) detects these discontinuities—particularly the presence of a CIDToGIDMap—and rewrites the mapping so that sequential CIDs 1..N correctly correspond to their Unicode values. This remapping occurs before the final CMap merge, ensuring that subsetted fonts decode accurately without requiring the original, complete typeface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}