How pdf-inspector Detects Broken Font Encodings: Three Heuristics Explained

The pdf-inspector library flags broken font encodings by detecting U+FFFD replacement characters, dollar-as-space patterns, and substitution-cipher garbling in extracted PDF text.

PDF text extraction fails silently when a font's ToUnicode CMap is missing, corrupted, or uses an uninterpretable legacy encoding. The open-source firecrawl/pdf-inspector repository solves this with a dedicated encoding-issues flag that triggers automatic OCR fallback. This article breaks down exactly how the detection works, with source code references from src/text_quality.rs.

---

What "Broken Font Encodings" Actually Means

PDFs store glyphs as internal byte codes. To extract readable text, these codes must map to Unicode through a font's CMap (character map). When this mapping breaks, you get:

• Gibberish like 8VceZWZTReV instead of "Certificate"
• Dollar signs between words: Word$Word$Word
• The Unicode replacement character (�) scattered through text

The pdf-inspector codebase treats such pages as untrustworthy and routes them to OCR. The detection logic lives in detect_encoding_issues at lines 31-50 of src/text_quality.rs.

---

Three Heuristics That Flag Encoding Failures

1. U+FFFD Replacement Characters — Direct Decode Failures

The simplest signal is the presence of U+FFFD (�), Unicode's "I don't know this character" placeholder.

In [src/text_quality.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L41-L44), lines 41-44:

if markdown.contains('\u{FFFD}') {
    return true;
}

This catches cases where the extractor cannot map any byte code to a valid Unicode point. One replacement character is enough to flag the entire page.

---

2. Dollar-as-Space Pattern — Broken ToUnicode CMaps

A common CMap corruption artifact is $ appearing between alphabetic characters where spaces should be, like Contract$Agreement$Section.

The has_dollar_as_space_pattern function (lines 57-73) triggers when either:

• More than 50% of $ characters appear in letter-$-letter positions, or
• More than 20 such occurrences exist on a page

This heuristic specifically targets malformed ToUnicode tables that misuse dollar signs as word separators.

---

3. Substitution-Cipher Garbling — Statistical Letter Analysis

Some broken CMaps shift every character by a constant offset, producing readable-looking but completely wrong text. The CipherGarbleStats struct evaluates this through statistical fingerprinting.

In CipherGarbleStats::looks_garbled, lines 85-124, the code:

1. Builds an ASCII letter histogram
2. Computes vowel ratio against expected English distributions
3. Measures case-shift rate (abnormal uppercase/lowercase patterns)
4. Calculates frequency-shape cosine similarity to detect shifted alphabets

A page fails this check when its letter statistics deviate significantly from natural language patterns.

---

Additional Encoding Warning Signals

Beyond the three core heuristics, src/text_quality.rs flags two more anomalies:

Signal	Function	Location	Meaning
Private-use characters	has_private_use_text_run	L78-L84	Unmapped glyphs falling into Unicode private-use areas
C1 control tokens	has_cid_control_token	L6-L9	Corrupted CID (Character ID) values leaking into output text

These act as secondary confirmation that font encoding resolution failed.

---

How the Flag Drives OCR Fallback

The encoding-issues detection feeds directly into pdf-inspector's quality-driven processing pipeline. In [src/lib.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs#L442-L450), lines 442-450, the process_pdf_with_options function consults has_encoding_issues:

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::process_mode::ProcessOptions;

// Run extraction with automatic quality detection
let opts = ProcessOptions::default();
let result = process_pdf_with_options("sample.pdf", opts)?;

// Check if broken font encodings triggered OCR fallback
if result.text_quality.has_encoding_issues {
    println!("⚠️  Broken font encodings detected – OCR used for affected pages");
}

When true, the system discards the extracted text for that page and reprocesses with OCR. This ensures output quality without manual intervention.

---

Manual Encoding Detection

You can invoke the low-level check directly on any markdown string:

use pdf_inspector::text_quality::detect_encoding_issues;

let suspicious_text = "The quick $brown$fox jumps over the lazy dog.";
if detect_encoding_issues(suspicious_text) {
    println!("Dollar-as-space pattern detected → broken encoding likely");
}

This is useful for debugging or integrating pdf-inspector's detection into custom pipelines.

---

Summary

• U+FFFD detection catches direct Unicode decode failures in src/text_quality.rs lines 41-44
• Dollar-as-space pattern identifies broken ToUnicode CMaps at lines 57-73
• CipherGarbleStats uses statistical analysis to detect substitution-cipher garbling at lines 85-124
• Private-use characters and C1 control tokens provide secondary confirmation
• The has_encoding_issues boolean in TextQualityReport triggers automatic OCR fallback in src/lib.rs

---

Frequently Asked Questions

What causes broken font encodings in PDFs?

Missing or corrupted ToUnicode CMaps are the primary cause. PDFs created by older software, scanned documents with hidden text layers, or files with embedded subset fonts that strip mapping tables commonly exhibit this problem. Legacy encodings like WinAnsi or MacRoman without proper Unicode conversion also trigger detection.

Can pdf-inspector fix broken encodings automatically?

No. The library detects broken encodings but cannot repair them—font encoding damage requires OCR fallback to recover readable text. The detection ensures OCR runs only when necessary, preserving extraction speed for well-formed documents.

How accurate is the substitution-cipher heuristic?

The CipherGarbleStats analysis has low false-positive rates for English text due to its multi-factor scoring (vowel ratio, case patterns, frequency shape). However, valid text with unusual character distributions—such as mathematical notation or code snippets—may occasionally trigger flags. The system prioritizes recall over precision since OCR fallback is safe.

Does the encoding-issues flag work for non-English languages?

The statistical heuristics are tuned for Latin-script languages. CJK (Chinese, Japanese, Korean) documents with broken encodings rely more heavily on U+FFFD detection and private-use character flagging, as the letter-frequency analysis assumes ASCII-range distributions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}