# How pdf-inspector Detects Encoding Issues and Triggers OCR Fallback

> Learn how pdf-inspector uses advanced heuristics like Unicode replacement and statistical analysis to detect encoding errors and trigger OCR fallback for corrupted pages.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: how-to-guide
- Published: 2026-08-14

---

**pdf-inspector uses multi-level heuristics—Unicode replacement characters, dollar-as-space patterns, and statistical frequency analysis—to flag broken font encodings and automatically route corrupted pages to OCR.**

Detecting encoding issues in PDF text extraction is critical for reliable document processing. The pdf-inspector library from Firecrawl implements a sophisticated two-tier detection system that identifies when a PDF's text layer cannot be trusted, triggering targeted OCR fallback only where needed. This approach avoids the performance cost of running OCR on every page while ensuring no garbled text slips through.

## Two-Level Detection Architecture

pdf-inspector examines text quality at both the **markdown level** (final assembled output) and the **span level** (individual text fragments before assembly). Both levels run identical heuristics implemented in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs).

| Level | Examination Target | Primary Use Case |
|-------|-------------------|------------------|
| Markdown-level | Final markdown string per page/region | Page-wide OCR decisions |
| Span-level | Individual `TextItem` spans | Early filtering during extraction |

The core detection logic lives in [`detect_encoding_issues`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs#L31) at line 31 of **src/text_quality.rs**, with span-level checks in `text_span_decoding_issue_kind` around lines 14–30.

## Three Encoding Issue Heuristics

### 1. Unicode Replacement Character (U+FFFD)

The presence of `U+FFFD` indicates a decode failure where a character from the PDF's font encoding could not be mapped to Unicode. This is the strongest signal of a broken **ToUnicode** CMap.

### 2. Dollar-as-Space Pattern

Broken CMaps sometimes substitute the `$` glyph for normal spaces, producing patterns like `Word$Word$Word`. This pattern is extremely rare in natural text and gets detected by `has_dollar_as_space_pattern`.

### 3. Statistical Frequency Analysis

The `CipherGarbleStats` implementation compares ASCII-letter frequency distributions to normal English. Substitution-cipher garbling (e.g., "Certificate" → "8VceZWZTReV") preserves letter-shape statistics but produces low cosine similarity to expected frequencies.

```rust
// Example: Manual encoding check on extracted markdown
use pdf_inspector::text_quality::detect_encoding_issues;

let markdown = "Hello $World$from$PDF";
if detect_encoding_issues(markdown) {
    println!("Encoding issues detected – fall back to OCR");
}

```

## OCR Fallback Trigger Pipeline

When any heuristic returns **true**, the pipeline executes three steps:

### Step 1: Record OCR Reason

The system assigns `OCR_REASON_SUSPECTED_GARBLED_TEXT` (constant value `"suspected_garbled_text"`) via `add_ocr_reason`. This occurs in `extract_pages_markdown_mem` around line 1000 of **src/lib.rs**.

### Step 2: Propagate Flags to Results

Two key result fields are populated in **src/lib.rs** around line 155:

- `PdfProcessResult.has_encoding_issues` — boolean flag if any page required OCR
- `pages_needing_ocr` — vector of 1-indexed page numbers

### Step 3: Enable Targeted OCR

The caller receives the page list and reason, invoking OCR only for affected pages.

```rust
// Example: High-level API with encoding-aware results
use pdf_inspector::{process_pdf, OCR_REASON_SUSPECTED_GARBLED_TEXT};

let result = process_pdf("sample.pdf").unwrap();

if result.has_encoding_issues {
    for page in result.pages_needing_ocr {
        println!("Page {} needs OCR ({})", page, OCR_REASON_SUSPECTED_GARBLED_TEXT);
    }
}

```

## Region-Based Extraction Support

For `extract_text_in_regions_mem`, the detection runs per-region. Around lines 6900–6920 of **src/lib.rs**, the code combines three quality checks:

```rust
let has_cid_issue = is_cid_garbage(&text);
let has_encoding_issue = detect_encoding_issues(&text);
let ocr_reason = if has_text_quality_issue || has_cid_issue || has_encoding_issue {
    Some(suspected_garbled_reason())
} else {
    None
};

```

If any check fires, `region.needs_ocr` becomes `true` with the attached reason.

```rust
// Example: Region extraction with automatic OCR fallback flags
use pdf_inspector::extract_text_in_regions_mem;

let buffer = std::fs::read("sample.pdf").unwrap();
let regions = vec![(0, vec![[50.0, 50.0, 300.0, 200.0]])];
let results = extract_text_in_regions_mem(&buffer, &regions).unwrap();

for reg in results[0].regions.iter() {
    if reg.needs_ocr {
        println!("Region needs OCR: {:?}", reg.ocr_reason);
    } else {
        println!("Clean text: {}", reg.text);
    }
}

```

## Key Source Files and Functions

| File | Key Components |
|------|---------------|
| **src/text_quality.rs** | `detect_encoding_issues`, `has_dollar_as_space_pattern`, `CipherGarbleStats` |
| **src/lib.rs** | `extract_pages_markdown_mem`, `add_ocr_reason`, `PdfProcessResult.has_encoding_issues` |
| **src/extractor/mod.rs** | `TextItem` span generation for span-level checks |

## Summary

- pdf-inspector detects encoding issues through **three complementary heuristics**: U+FFFD, dollar-as-space patterns, and statistical letter-frequency analysis
- Detection operates at **both markdown and span levels** for comprehensive coverage
- Flagged content triggers **targeted OCR fallback** via `OCR_REASON_SUSPECTED_GARBLED_TEXT` without processing clean pages
- The architecture supports **page-wide and region-specific** extraction modes
- Core implementation resides in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) with orchestration in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)

## Frequently Asked Questions

### What causes the U+FFFD replacement character in PDF extraction?

U+FFFD appears when the PDF's font encoding cannot be mapped to Unicode—typically due to a missing or malformed **ToUnicode** CMap in the font dictionary. pdf-inspector treats this as definitive evidence of encoding failure.

### How does pdf-inspector avoid unnecessary OCR processing?

The library only marks specific pages or regions with `needs_ocr = true`. Callers receive precise page lists through `pages_needing_ocr`, enabling **selective OCR invocation** rather than document-wide processing.

### Can I use encoding detection without the full pdf-inspector pipeline?

Yes. The `detect_encoding_issues` function in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) is **publicly exposed** and can be imported directly for custom text quality checks on any string.

### Why combine three heuristics instead of using U+FFFD alone?

U+FFFD catches explicit decode failures, but **substitution ciphers** and **glyph misassignments** (like dollar-as-space) produce valid Unicode that renders incorrectly. The statistical and pattern-based heuristics catch these **silent corruptions** that pure decode-error detection would miss.