# How pdf-inspector Detects Broken Font Encodings: Three Heuristics Explained

> Learn how pdf-inspector detects broken font encodings using three heuristics: replacement characters, dollar-as-space patterns, and substitution-cipher garbling. Improve your PDF text extraction.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-09-02

---

**The pdf-inspector library flags broken font encodings by detecting U+FFFD replacement characters, dollar-as-space patterns, and substitution-cipher garbling in extracted PDF text.**

PDF text extraction fails silently when a font's **ToUnicode CMap** is missing, corrupted, or uses an uninterpretable legacy encoding. The open-source `firecrawl/pdf-inspector` repository solves this with a dedicated `encoding-issues` flag that triggers automatic OCR fallback. This article breaks down exactly how the detection works, with source code references from [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs).

---

## What "Broken Font Encodings" Actually Means

PDFs store glyphs as internal byte codes. To extract readable text, these codes must map to Unicode through a font's **CMap** (character map). When this mapping breaks, you get:

- Gibberish like `8VceZWZTReV` instead of "Certificate"
- Dollar signs between words: `Word$Word$Word`
- The Unicode replacement character (�) scattered through text

The `pdf-inspector` codebase treats such pages as untrustworthy and routes them to OCR. The detection logic lives in [`detect_encoding_issues`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L31-L50) at lines 31-50 of [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs).

---

## Three Heuristics That Flag Encoding Failures

### 1. U+FFFD Replacement Characters — Direct Decode Failures

The simplest signal is the presence of `U+FFFD` (�), Unicode's "I don't know this character" placeholder.

In [[`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs)](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L41-L44), lines 41-44:

```rust
if markdown.contains('\u{FFFD}') {
    return true;
}

```

This catches cases where the extractor cannot map any byte code to a valid Unicode point. One replacement character is enough to flag the entire page.

---

### 2. Dollar-as-Space Pattern — Broken ToUnicode CMaps

A common CMap corruption artifact is `$` appearing between alphabetic characters where spaces should be, like `Contract$Agreement$Section`.

The [`has_dollar_as_space_pattern`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L57-L73) function (lines 57-73) triggers when either:

- More than 50% of `$` characters appear in **letter-$-letter** positions, **or**
- More than 20 such occurrences exist on a page

This heuristic specifically targets malformed **ToUnicode** tables that misuse dollar signs as word separators.

---

### 3. Substitution-Cipher Garbling — Statistical Letter Analysis

Some broken CMaps shift every character by a constant offset, producing readable-looking but completely wrong text. The `CipherGarbleStats` struct evaluates this through statistical fingerprinting.

In [`CipherGarbleStats::looks_garbled`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L85-L124), lines 85-124, the code:

1. Builds an ASCII letter histogram
2. Computes **vowel ratio** against expected English distributions
3. Measures **case-shift rate** (abnormal uppercase/lowercase patterns)
4. Calculates **frequency-shape cosine similarity** to detect shifted alphabets

A page fails this check when its letter statistics deviate significantly from natural language patterns.

---

## Additional Encoding Warning Signals

Beyond the three core heuristics, [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) flags two more anomalies:

| Signal | Function | Location | Meaning |
|--------|----------|----------|---------|
| **Private-use characters** | `has_private_use_text_run` | [L78-L84](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L78-L84) | Unmapped glyphs falling into Unicode private-use areas |
| **C1 control tokens** | `has_cid_control_token` | [L6-L9](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L6-L9) | Corrupted CID (Character ID) values leaking into output text |

These act as secondary confirmation that font encoding resolution failed.

---

## How the Flag Drives OCR Fallback

The `encoding-issues` detection feeds directly into pdf-inspector's quality-driven processing pipeline. In [[`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs#L442-L450), lines 442-450, the `process_pdf_with_options` function consults `has_encoding_issues`:

```rust
use pdf_inspector::process_pdf_with_options;
use pdf_inspector::process_mode::ProcessOptions;

// Run extraction with automatic quality detection
let opts = ProcessOptions::default();
let result = process_pdf_with_options("sample.pdf", opts)?;

// Check if broken font encodings triggered OCR fallback
if result.text_quality.has_encoding_issues {
    println!("⚠️  Broken font encodings detected – OCR used for affected pages");
}

```

When `true`, the system discards the extracted text for that page and reprocesses with OCR. This ensures output quality without manual intervention.

---

## Manual Encoding Detection

You can invoke the low-level check directly on any markdown string:

```rust
use pdf_inspector::text_quality::detect_encoding_issues;

let suspicious_text = "The quick $brown$fox jumps over the lazy dog.";
if detect_encoding_issues(suspicious_text) {
    println!("Dollar-as-space pattern detected → broken encoding likely");
}

```

This is useful for debugging or integrating pdf-inspector's detection into custom pipelines.

---

## Summary

- **U+FFFD detection** catches direct Unicode decode failures in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) lines 41-44
- **Dollar-as-space pattern** identifies broken ToUnicode CMaps at lines 57-73
- **CipherGarbleStats** uses statistical analysis to detect substitution-cipher garbling at lines 85-124
- Private-use characters and C1 control tokens provide secondary confirmation
- The `has_encoding_issues` boolean in `TextQualityReport` triggers automatic OCR fallback in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)

---

## Frequently Asked Questions

### What causes broken font encodings in PDFs?

Missing or corrupted **ToUnicode CMaps** are the primary cause. PDFs created by older software, scanned documents with hidden text layers, or files with embedded subset fonts that strip mapping tables commonly exhibit this problem. Legacy encodings like **WinAnsi** or **MacRoman** without proper Unicode conversion also trigger detection.

### Can pdf-inspector fix broken encodings automatically?

No. The library **detects** broken encodings but cannot repair them—font encoding damage requires OCR fallback to recover readable text. The detection ensures OCR runs only when necessary, preserving extraction speed for well-formed documents.

### How accurate is the substitution-cipher heuristic?

The `CipherGarbleStats` analysis has low false-positive rates for English text due to its multi-factor scoring (vowel ratio, case patterns, frequency shape). However, valid text with unusual character distributions—such as mathematical notation or code snippets—may occasionally trigger flags. The system prioritizes **recall over precision** since OCR fallback is safe.

### Does the encoding-issues flag work for non-English languages?

The statistical heuristics are tuned for Latin-script languages. CJK (Chinese, Japanese, Korean) documents with broken encodings rely more heavily on U+FFFD detection and private-use character flagging, as the letter-frequency analysis assumes ASCII-range distributions.