How pdf-inspector's Font Encoding Fallback Mechanism Decodes PDFs Without ToUnicode CMaps

pdf-inspector extracts text by first attempting to parse the embedded ToUnicode CMap, then falling back through encoding-based, TrueType cmap table, and simple-font strategies when that CMap is missing, corrupted, or too sparse.

The firecrawl/pdf-inspector library implements a robust, multi-layered fallback mechanism for font encoding in PDF text extraction. When a PDF lacks a usable ToUnicode CMap—the standard mapping from character codes to Unicode—pdf-inspector reconstructs character maps from the font's intrinsic encoding data. This fallback system is fully implemented in src/tournicode.rs and ensures reliable text recovery even from poorly encoded documents.

How the Primary ToUnicode Parsing Works

The extraction process begins with the embedded ToUnicode CMap. This is the standard PDF mechanism for mapping internal character codes to Unicode code points.

if let Some(cmap) = ToUnicodeCMap::parse(data) { … }

When this parsing succeeds, the resulting map becomes the primary character map for decoding. However, pdf-inspector applies a sparsity threshold to determine if the primary map is actually usable.

The Sparse-Map Threshold

If the parsed CMap contains fewer than 10 entries, it is considered too sparse to provide meaningful character coverage:

if primary_entries < 10 { … }

This threshold triggers the fallback chain described below.

The Three-Tier Fallback Chain in pdf-inspector

When the primary ToUnicode CMap fails or is too sparse, pdf-inspector executes three independent fallback generators in sequence. Each returns an Option<ToUnicodeCMap>:

1. Encoding-Based Fallback

The first fallback builds a character map from the font's built-in encoding such as WinAnsi or MacRoman:

.or_else(|| build_fallback_tounicode_from_encoding(font_dict, doc))

This strategy works for standard PDF fonts that declare their encoding explicitly but lack a ToUnicode CMap.

2. Type-0 (TrueType) Fallback

For CID-based Type-0 fonts, pdf-inspector parses the embedded TrueType cmap table to reconstruct the character map:

.or_else(|| build_fallback_cmap_for_type0(font_dict, doc))

This fallback is particularly effective for modern PDFs embedding TrueType or OpenType fonts with rich Unicode coverage in their cmap tables.

3. Simple-Font Fallback

The final fallback handles simple (non-CID) fonts by using their embedded font cmap or glyph-name table:

.or_else(|| build_fallback_cmap_for_simple(font_dict, doc))

These three calls appear contiguously at lines 40–43 of src/tounicode.rs in the pdf-inspector source code.

Fallback Promotion: Selecting the Best Character Map

After a fallback is selected, pdf-inspector applies a promotion logic that compares map completeness. A more comprehensive fallback—such as a TrueType cmap with 45 entries—can replace a sparse primary map with only 3 entries:

// promotion logic shown at lines 61-73 of src/tounicode.rs

The most expressive mapping becomes the primary map, while the less complete one is retained as a secondary fallback. This ensures optimal text decoding quality.

Final Fallback for Completely Missing CMaps

If the primary ToUnicode CMap cannot be parsed at all, pdf-inspector bypasses sparsity checks and directly executes the Type-0 or simple-font strategies (lines 83–90). The resulting map serves as the sole source of character decoding.

Practical Example: Triggering Fallback Behavior

You can observe the fallback mechanism in action using pdf-inspector's public API. The process_pdf_with_options function in src/lib.rs orchestrates the entire extraction pipeline:

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::process_mode::ProcessMode;

fn main() {
    // Load a PDF that has no ToUnicode CMap.
    let pdf_bytes = std::fs::read("samples/no_tounicode.pdf").unwrap();

    // Process with default options (fallbacks are enabled).
    let result = process_pdf_with_options(pdf_bytes, ProcessMode::default());

    println!("{}", result.markdown);
}

When processing no_tounicode.pdf, pdf-inspector:

1. Attempts ToUnicode CMap parsing → failure
2. Executes the fallback chain (encoding → Type-0 → simple)
3. Produces structured markdown with correctly decoded characters

Debugging Fallback Selection

Enable debug logging to trace which fallback strategy was applied:

RUST_LOG=pdf_inspector::tounicode=debug cargo run --bin pdf2md -- samples/no_tounicode.pdf

Expected log output:

ToUnicode CMap obj=5 too sparse (3 entries); using fallback
ToUnicode CMap obj=5: TrueType fallback (45 entries) > primary (3); promoting over sequential remap

These messages correspond to lines 46–53 and 61–73 of src/tournicode.rs in pdf-inspector.

Key Source Files in pdf-inspector

File	Purpose
src/tournicode.rs	Core ToUnicode parsing and fallback generation implementation
src/lib.rs	Public API (process_pdf_with_options) that triggers CMap logic
src/extractor/fonts.rs	Font-specific utilities supporting fallback strategies
src/detector.rs	PDF type detection that may trigger OCR fallback

Summary

• Primary strategy: Parse embedded ToUnicode CMap; reject if < 10 entries
• Fallback chain: Encoding-based → TrueType cmap table → simple-font glyph tables
• Promotion logic: More complete fallbacks overwrite sparse primary maps
• API entry point: process_pdf_with_options in src/lib.rs
• Debug visibility: RUST_LOG=pdf_inspector::tounicode=debug exposes fallback decisions

Frequently Asked Questions

What triggers the font encoding fallback in pdf-inspector?

The fallback activates when the embedded ToUnicode CMap is missing, corrupt, or contains fewer than 10 entries. This sparsity threshold at line 46 of src/tournicode.rs ensures that incomplete maps don't produce garbled text.

Does pdf-inspector support Chinese, Japanese, or Korean PDFs without ToUnicode CMaps?

Yes, through the Type-0 fallback (build_fallback_cmap_for_type0). This strategy parses the TrueType cmap table embedded in CID-based fonts, which often contains comprehensive Unicode mappings for CJK characters independent of the PDF's ToUnicode data.

Can developers customize or disable the fallback mechanism?

The public API (process_pdf_with_options) does not expose direct fallback toggles. However, fallback behavior is always enabled in the current implementation. To modify fallback logic, you would need to fork and edit the source in src/tournicode.rs—specifically the chain at lines 40–43.

How does pdf-inspector's fallback differ from other PDF extraction tools?

Many tools either fail entirely or produce raw character codes when ToUnicode is absent. pdf-inspector's three-tier progressive fallback extracts usable text by progressively leveraging richer font data sources—encoding dictionaries, TrueType tables, and glyph names—which increases recovery rates for legacy or poorly generated PDFs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}