# How pdf-inspector Handles CMap and ToUnicode Parsing for Font Decoding

> Discover how pdf-inspector decodes fonts using a four-tier fallback strategy for CMap and ToUnicode parsing. Learn about CID to Unicode conversion in our latest technical breakdown.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-08-13

---

**pdf-inspector** implements a four-tier fallback strategy in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) that converts PDF character identifiers (CIDs) into Unicode strings by parsing `/ToUnicode` streams, binary CMaps, built-in encodings, TrueType tables, and glyph names.

Extracting readable text from PDFs requires decoding font-specific character identifiers into Unicode, a process governed by **CMap and ToUnicode parsing**. The pdf-inspector library, developed by Firecrawl as a Rust-based text extraction engine, solves this through a robust pipeline implemented primarily in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs). This system handles everything from standard ASCII CMaps to compact binary formats and subset font remapping, ensuring accurate text extraction even from malformed or strictly encoded documents.

## The Core Architecture

The font decoding system centers on two primary modules. The [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) file contains the low-level parsing logic for CMap structures, while [`src/extractor/fonts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/fonts.rs) orchestrates the **FontCMaps** structure that manages the layered fallback strategy. When processing a PDF page, the extractor builds a `FontCMaps` instance for each font object, which attempts to acquire the most complete Unicode mapping available through a prioritized sequence of sources.

The pipeline handles both **ASCII-based** CMap syntax and compact **binary CMap** formats (such as Adobe-Japan1), detecting the format automatically during the parse phase. For each text-showing operator encountered in the content stream, the system calls `ToUnicodeCMap::decode` or `lookup_cid` to transform raw byte sequences into valid Unicode strings.

## The CMap Parsing Pipeline

### Loading the ToUnicode Stream

For every font object, pdf-inspector first searches for a `/ToUnicode` entry in the font dictionary. When present, the raw stream bytes are passed to `ToUnicodeCMap::parse` (lines 103-130 in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs)), which interprets the PDF CMap syntax regardless of whether it uses ASCII or binary encoding. This function serves as the primary entry point for character mapping discovery.

### Handling Binary CMap Formats

Many professional PDFs embed compact binary CMaps to reduce file size. The `parse_binary_cmap` function (lines 1244-1320) processes these streams through a `BinaryCMapStream` reader, extracting the code-byte length, character-to-Unicode ranges, and populating internal `char_map` and `ranges` tables. This allows the library to decode complex CJK (Chinese, Japanese, Korean) fonts that rely on predefined binary mappings.

### Built-in CMap Fallbacks

When the `/ToUnicode` stream is missing or contains sparse entries, pdf-inspector falls back to **pre-built CMaps** shipped with the binary. The `load_builtin_cmap_by_name` function (lines 1870-1885) retrieves standardized maps such as `Adobe-Korea1` or `Adobe-Japan1` from embedded resources. These built-in maps undergo the same parsing pipeline as binary streams, ensuring consistent behavior across all CMap sources.

## Fallback Strategies for Robust Decoding

If the primary ToUnicode mapping proves insufficient, pdf-inspector activates a sequence of tertiary extraction methods designed to handle edge cases in PDF generation.

### TrueType cmap Table Extraction

When ToUnicode data is unavailable, the library probes embedded TrueType fonts via `/FontDescriptor/FontFile2`. The `build_cmap_from_truetype` function (lines 967-990) uses `ttf_parser` to read the font's internal `cmap` table, constructing a CID-to-Unicode map directly from glyph IDs. This approach recovers text from PDFs generated by applications that neglect to include explicit ToUnicode CMaps.

### Adobe Glyph List Resolution

Some fonts expose only glyph names through the **post** table rather than explicit Unicode mappings. The `build_cmap_from_glyph_names` function (lines 1134-1150) maps these names to Unicode using the **Adobe Glyph List (AGL)**, enabling text extraction from symbolic or decorative fonts that lack standard encoding tables.

### Subset Font Remapping

PDF generators frequently subset fonts, causing the CID space to become non-sequential. The `remap_to_sequential` function (lines 548-566) detects these mismatches—particularly when a `CIDToGIDMap` is present—and rewrites the CMap so that CIDs 1 through N correctly map to their corresponding Unicode values. This correction is essential for documents using embedded subsets of larger typefaces.

### Merging Multiple CMap Sources

Complex fonts may contain several CMap variants simultaneously. The `merge_cmaps` function (lines 1883-1898) combines primary, remapped, and fallback mappings into a unified structure, preferring the most complete mapping while preserving alternatives for error recovery. This ensures that even partially corrupt CMaps can contribute valid decoding information.

## Text Decoding in Content Streams

During PDF content-stream processing in [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs), each text operator (`Tj`, `TJ`, `Td`, `Tm`) emits raw byte sequences. The extractor retrieves the appropriate `FontCMaps` entry for the current font, then invokes `ToUnicodeCMap::decode` (lines 450-460) or `lookup_cid` to convert these bytes into Unicode strings. This decoded text feeds directly into the markdown conversion pipeline, preserving the document's logical reading order and character set.

## Practical Implementation Examples

### High-Level PDF Text Extraction

```rust
use pdf_inspector::process_pdf_with_options;
use pdf_inspector::ExtractionOptions;

let options = ExtractionOptions::default();
let markdown = process_pdf_with_options("sample.pdf", options).unwrap();
println!("{}", markdown);

```

This API automatically builds `FontCMaps` for each detected font and applies the full decoding pipeline during extraction.

### Manual CMap Inspection

```rust
use pdf_inspector::extractor::fonts::FontCMaps;
use lopdf::Document;

let doc = Document::load("sample.pdf")?;
let font_dicts = /* collect Font objects from the page */;
let cmap_collection = FontCMaps::build(&doc, &font_dicts, None, false)?;

// Inspect the primary CMap of the first font
let entry = cmap_collection.by_obj_num.values().next().unwrap();
println!("Primary entries: {}", entry.primary.char_map.len());
println!("Remapped? {}", entry.remapped.is_some());

```

### Raw CID Sequence Decoding

```rust
use pdf_inspector::tounicode::ToUnicodeCMap;

// Assuming a parsed CMap `cmap`
let raw_bytes = vec![0x00, 0x41]; // CID for "A" in a 2-byte CMap
let text = cmap.decode(&raw_bytes).unwrap();
assert_eq!(text, "A");

```

## Summary

- **pdf-inspector** implements a four-tier fallback strategy: ToUnicode CMaps, built-in CMaps, TrueType tables, and glyph-name mapping.
- The core parsing logic resides in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs), with `ToUnicodeCMap::parse` handling both ASCII and binary CMap formats.
- **Binary CMap support** enables proper decoding of CJK fonts through `parse_binary_cmap` and `BinaryCMapStream`.
- **Subset font handling** via `remap_to_sequential` corrects non-sequential CID spaces common in optimized PDFs.
- The architecture combines these sources through `merge_cmaps` to maximize text recovery from malformed or minimally encoded documents.

## Frequently Asked Questions

### What is a ToUnicode CMap in PDF processing?

A **ToUnicode CMap** is a PDF stream object that maps character identifiers (CIDs) used in content streams to standardized Unicode code points. According to the pdf-inspector source code, this mapping is essential because PDFs often use custom font encodings where the byte value `0x41` might not represent the Latin letter "A". The `ToUnicodeCMap::parse` function in [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) handles both ASCII and binary representations of these mappings.

### How does pdf-inspector handle binary versus ASCII CMap streams?

The library automatically detects the format during parsing. ASCII CMaps are parsed directly by the `ToUnicodeCMap::parse` function, while binary CMaps are routed to `parse_binary_cmap` (lines 1244-1320), which uses a specialized `BinaryCMapStream` reader to extract code lengths and character ranges. Both formats ultimately populate the same internal `char_map` structures for consistent lookup performance.

### What happens when a PDF font lacks a ToUnicode entry?

When the `/ToUnicode` stream is missing or sparse, pdf-inspector activates its fallback chain. First, it attempts to load **built-in CMaps** via `load_builtin_cmap_by_name`. If that fails, it probes the embedded TrueType font's `cmap` table using `build_cmap_from_truetype`. As a final resort, it uses `build_cmap_from_glyph_names` to map glyph names to Unicode via the Adobe Glyph List. The system only skips a font when all four methods fail to produce viable mappings.

### How does the library handle subsetted fonts with non-sequential CIDs?

PDF generators often subset fonts by remapping glyph indices, causing CIDs to skip values or appear out of order. The `remap_to_sequential` function (lines 548-566) detects these discontinuities—particularly the presence of a `CIDToGIDMap`—and rewrites the mapping so that sequential CIDs 1..N correctly correspond to their Unicode values. This remapping occurs before the final CMap merge, ensuring that subsetted fonts decode accurately without requiring the original, complete typeface.