How pdf-inspector Detects Encoding Issues and Triggers OCR Fallback

pdf-inspector uses multi-level heuristics—Unicode replacement characters, dollar-as-space patterns, and statistical frequency analysis—to flag broken font encodings and automatically route corrupted pages to OCR.

Detecting encoding issues in PDF text extraction is critical for reliable document processing. The pdf-inspector library from Firecrawl implements a sophisticated two-tier detection system that identifies when a PDF's text layer cannot be trusted, triggering targeted OCR fallback only where needed. This approach avoids the performance cost of running OCR on every page while ensuring no garbled text slips through.

Two-Level Detection Architecture

pdf-inspector examines text quality at both the markdown level (final assembled output) and the span level (individual text fragments before assembly). Both levels run identical heuristics implemented in src/text_quality.rs.

Level	Examination Target	Primary Use Case
Markdown-level	Final markdown string per page/region	Page-wide OCR decisions
Span-level	Individual TextItem spans	Early filtering during extraction

The core detection logic lives in detect_encoding_issues at line 31 of src/text_quality.rs, with span-level checks in text_span_decoding_issue_kind around lines 14–30.

Three Encoding Issue Heuristics

1. Unicode Replacement Character (U+FFFD)

The presence of U+FFFD indicates a decode failure where a character from the PDF's font encoding could not be mapped to Unicode. This is the strongest signal of a broken ToUnicode CMap.

2. Dollar-as-Space Pattern

Broken CMaps sometimes substitute the $ glyph for normal spaces, producing patterns like Word$Word$Word. This pattern is extremely rare in natural text and gets detected by has_dollar_as_space_pattern.

3. Statistical Frequency Analysis

The CipherGarbleStats implementation compares ASCII-letter frequency distributions to normal English. Substitution-cipher garbling (e.g., "Certificate" → "8VceZWZTReV") preserves letter-shape statistics but produces low cosine similarity to expected frequencies.

// Example: Manual encoding check on extracted markdown
use pdf_inspector::text_quality::detect_encoding_issues;

let markdown = "Hello $World$from$PDF";
if detect_encoding_issues(markdown) {
    println!("Encoding issues detected – fall back to OCR");
}

OCR Fallback Trigger Pipeline

When any heuristic returns true, the pipeline executes three steps:

Step 1: Record OCR Reason

The system assigns OCR_REASON_SUSPECTED_GARBLED_TEXT (constant value "suspected_garbled_text") via add_ocr_reason. This occurs in extract_pages_markdown_mem around line 1000 of src/lib.rs.

Step 2: Propagate Flags to Results

Two key result fields are populated in src/lib.rs around line 155:

• PdfProcessResult.has_encoding_issues — boolean flag if any page required OCR
• pages_needing_ocr — vector of 1-indexed page numbers

Step 3: Enable Targeted OCR

The caller receives the page list and reason, invoking OCR only for affected pages.

// Example: High-level API with encoding-aware results
use pdf_inspector::{process_pdf, OCR_REASON_SUSPECTED_GARBLED_TEXT};

let result = process_pdf("sample.pdf").unwrap();

if result.has_encoding_issues {
    for page in result.pages_needing_ocr {
        println!("Page {} needs OCR ({})", page, OCR_REASON_SUSPECTED_GARBLED_TEXT);
    }
}

Region-Based Extraction Support

For extract_text_in_regions_mem, the detection runs per-region. Around lines 6900–6920 of src/lib.rs, the code combines three quality checks:

let has_cid_issue = is_cid_garbage(&text);
let has_encoding_issue = detect_encoding_issues(&text);
let ocr_reason = if has_text_quality_issue || has_cid_issue || has_encoding_issue {
    Some(suspected_garbled_reason())
} else {
    None
};

If any check fires, region.needs_ocr becomes true with the attached reason.

// Example: Region extraction with automatic OCR fallback flags
use pdf_inspector::extract_text_in_regions_mem;

let buffer = std::fs::read("sample.pdf").unwrap();
let regions = vec![(0, vec![[50.0, 50.0, 300.0, 200.0]])];
let results = extract_text_in_regions_mem(&buffer, &regions).unwrap();

for reg in results[0].regions.iter() {
    if reg.needs_ocr {
        println!("Region needs OCR: {:?}", reg.ocr_reason);
    } else {
        println!("Clean text: {}", reg.text);
    }
}

Key Source Files and Functions

File	Key Components
src/text_quality.rs	detect_encoding_issues, has_dollar_as_space_pattern, CipherGarbleStats
src/lib.rs	extract_pages_markdown_mem, add_ocr_reason, PdfProcessResult.has_encoding_issues
src/extractor/mod.rs	TextItem span generation for span-level checks

Summary

• pdf-inspector detects encoding issues through three complementary heuristics: U+FFFD, dollar-as-space patterns, and statistical letter-frequency analysis
• Detection operates at both markdown and span levels for comprehensive coverage
• Flagged content triggers targeted OCR fallback via OCR_REASON_SUSPECTED_GARBLED_TEXT without processing clean pages
• The architecture supports page-wide and region-specific extraction modes
• Core implementation resides in src/text_quality.rs with orchestration in src/lib.rs

Frequently Asked Questions

What causes the U+FFFD replacement character in PDF extraction?

U+FFFD appears when the PDF's font encoding cannot be mapped to Unicode—typically due to a missing or malformed ToUnicode CMap in the font dictionary. pdf-inspector treats this as definitive evidence of encoding failure.

How does pdf-inspector avoid unnecessary OCR processing?

The library only marks specific pages or regions with needs_ocr = true. Callers receive precise page lists through pages_needing_ocr, enabling selective OCR invocation rather than document-wide processing.

Can I use encoding detection without the full pdf-inspector pipeline?

Yes. The detect_encoding_issues function in src/text_quality.rs is publicly exposed and can be imported directly for custom text quality checks on any string.

Why combine three heuristics instead of using U+FFFD alone?

U+FFFD catches explicit decode failures, but substitution ciphers and glyph misassignments (like dollar-as-space) produce valid Unicode that renders incorrectly. The statistical and pattern-based heuristics catch these silent corruptions that pure decode-error detection would miss.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}