# Handling PDF Encoding Issues and Garbled Text with pdf-inspector: A Complete Guide

> Fix garbled PDF text and encoding issues with pdf-inspector. Learn how this library detects problematic pages and uses OCR fallback for cleaner results.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: how-to-guide
- Published: 2026-08-10

---

**The pdf-inspector library automatically detects garbled PDF text using heuristic checks for replacement characters, substitution-cipher patterns, and CID-related garbage, then routes problematic pages to OCR fallback.**

PDFs often contain **broken font encodings** that produce unreadable output — missing *ToUnicode* CMaps, failed CID-to-Unicode mappings, or glyphs rendered as vector shapes. The `firecrawl/pdf-inspector` crate solves this through a multi-stage pipeline that isolates encoding problems at both the markdown and individual text-item level, enabling automatic OCR fallback when extraction quality degrades.

## How pdf-inspector Detects Encoding Failures

The library operates a **detection → extraction → markdown rendering** pipeline. During extraction, it processes low-level `TextItem` structures from the Lopdf parser. When font encoding breaks down, these items contain mangled byte sequences that manifest as replacement characters, control codes, or statistically improbable letter distributions.

pdf-inspector implements **two complementary detection layers**:

| Level | Checks Performed | Source Location |
|-------|-------------------|-----------------|
| **Markdown-level** | U+FFFD replacement characters; "dollar-as-space" patterns (`Word$Word`); substitution-cipher letter statistics; non-alphanumeric dominance | [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs): `detect_encoding_issues`, `is_garbage_text` |
| **Item-level** | Runs of replacement characters; private-use / C1-control runs; CID-related garbage; statistical substitution-cipher evidence | [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs): `analyze_text_quality`, `region_items_have_decoding_issue` |

Both layers feed into the same decision logic: when quality checks fail, the page or region receives an **OCR reason flag** (`OCR_REASON_SUSPECTED_GARBLED_TEXT`, `OCR_REASON_SCANNED`, or similar) that downstream code can use to trigger alternative processing.

## The Quality Detection Pipeline in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)

The public API orchestrates detection through `process_pdf_with_options`. The flow in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) works as follows:

1. **`process_pdf_with_options`** loads the PDF and delegates to `process_document`
2. **`process_document`** invokes `extract_pages_markdown_mem` (or region-specific variants)
3. For each extracted page, the code evaluates quality:

```rust
let has_decoding_issue = has_text_quality_issue
    || (!md.is_empty() && (is_cid_garbage(&md) || detect_encoding_issues(&md)));

```

When `has_decoding_issue` evaluates to `true`, the system calls `add_ocr_reason` to flag the page. The caller receives this signal in the result structure and can initiate OCR fallback.

## Text Quality Heuristics in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs)

The **text-quality module** implements concrete detection algorithms. Each heuristic targets a specific failure mode common in malformed PDFs:

- **`has_replacement_text_run`** — Detects contiguous runs of **U+FFFD** (Unicode replacement character), indicating bytes that could not be decoded to valid Unicode
- **`has_private_use_text_run` / `has_cid_control_token`** — Identifies **private-use area codepoints** and **C1 control characters** that leak through when CID font mappings fail
- **`has_dollar_as_space_pattern`** — Recognizes the distinctive `Word$Word` pattern where dollar signs substitute for spaces due to encoding table corruption
- **`CipherGarbleStats`** — Performs **statistical substitution-cipher detection** using letter frequency analysis, shape-cosine similarity, and case-shift bigram modeling
- **`is_garbage_text`** — Measures **alphanumeric vs. non-alphanumeric ratio** while ignoring markdown syntax, flagging outputs dominated by symbols and punctuation

The module aggregates evidence through `PageTextQualityEvidence` and makes final OCR decisions via `page_replacement_evidence_needs_ocr`, which weighs replacement character density, run length, and cipher-likeness against configurable thresholds.

## Detecting Garbled PDFs Automatically

The simplest integration processes a complete PDF and reports which pages require OCR:

```rust
use pdf_inspector::{process_pdf_with_options, PdfOptions, ProcessMode};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = PdfOptions::new()
        .mode(ProcessMode::Full);

    let result = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;

    println!("PDF type: {:?}", result.pdf_type);
    println!("Pages needing OCR: {:?}", result.pages_needing_ocr);
    
    if let Some(md) = result.markdown {
        println!("Extracted markdown (clean pages only):\n{md}");
    }
    Ok(())
}

```

The `pages_needing_ocr` field reveals encoding failures detected during extraction. Use `ProcessMode::DetectOnly` for faster analysis when you only need quality assessment without full markdown generation.

## Region-Level Encoding Detection

For targeted extraction — extracting text from specific coordinates rather than full pages — the library performs quality checks per-region:

```rust
use pdf_inspector::extract_text_in_regions_mem;

fn main() -> Result<(), pdf_inspector::PdfError> {
    // Page 0, region from (50,700) to (550,750) in PDF points
    let regions = vec![(0u32, vec![[50.0, 700.0, 550.0, 750.0]])];

    let buffer = std::fs::read("invoice.pdf")?;
    let page_results = extract_text_in_regions_mem(&buffer, &regions)?;

    for page in page_results {
        for (i, region) in page.regions.iter().enumerate() {
            if region.needs_ocr {
                println!("Region {i} on page {} needs OCR", page.page + 1);
            } else {
                println!("Region {i} text: {}", region.text);
            }
        }
    }
    Ok(())
}

```

Each `RegionText` exposes a **boolean `needs_ocr` flag** populated by the same quality heuristics used for full-page extraction. This enables fine-grained fallback: OCR only the garbled regions while preserving clean extracted text elsewhere.

### Using the Low-Level Quality API

For custom pipelines, invoke detection functions directly on markdown strings:

```rust
use pdf_inspector::text_quality::{
    detect_encoding_issues, 
    is_garbage_text, 
    is_cid_garbage
};

fn main() {
    let md = "----1-.-.-.___  --.-. .._ I_---.";
    
    println!("Has replacement chars? {}", md.contains('\u{FFFD}'));
    println!("Encoding issues? {}", detect_encoding_issues(md));
    println!("Garbage text? {}", is_garbage_text(md));
    println!("CID garbage? {}", is_cid_garbage(md));
}

```

This interface supports **proactive quality monitoring** — validate externally-sourced markdown or compare extraction engines against pdf-inspector's detection criteria.

## Integration with OCR Fallback

The encoding detection system produces **structured OCR reasons** that integrate cleanly with GPU-based OCR pipelines. When `add_ocr_reason` marks a page with `OCR_REASON_SUSPECTED_GARBLED_TEXT`, the resulting metadata includes:

- The specific heuristic that triggered (replacement runs, CID garbage, cipher detection)
- Confidence level derived from `PageTextQualityEvidence` aggregation
- Page or region coordinates for targeted reprocessing

This design separates **quality assessment** from **remediation strategy** — your application decides whether to use local OCR, cloud vision APIs, or manual review based on the signaled reason codes.

## Key Source Files for PDF Encoding Handling

| File | Purpose |
|------|---------|
| [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) | Public API entry points: `process_pdf`, `detect_pdf`, `extract_pages_markdown_mem`; OCR reason propagation |
| [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) | All quality heuristics: replacement detection, cipher statistics, garbage classification, evidence aggregation |
| [`src/markdown/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/mod.rs) | Markdown generation and layout complexity analysis; integrates quality checks into rendering pipeline |
| [`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs) | Lopdf content-stream parser producing `TextItem`s consumed by quality analysis |
| [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) | PDF-type classification including scan detection and OCR triggering |

## Summary

- pdf-inspector detects **PDF encoding issues** through markdown-level and item-level heuristics in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs)
- **Quality signals** include replacement characters (U+FFFD), private-use runs, CID garbage, substitution-cipher statistics, and symbol-to-text ratios
- The **`process_pdf_with_options`** API automatically flags pages needing OCR via `pages_needing_ocr` in the result structure
- **Region-level extraction** with `extract_text_in_regions_mem` enables targeted quality assessment for specific document areas
- OCR reasons (`OCR_REASON_SUSPECTED_GARBLED_TEXT`) provide structured metadata for downstream remediation decisions

## Frequently Asked Questions

### What causes garbled text in PDF extraction?

PDF font encoding breaks when **ToUnicode CMaps are missing**, **CID-to-Unicode mappings fail**, or **glyphs are drawn as vector paths** rather than encoded text. The binary font data inside PDFs often uses custom encodings that mapping tables should convert to Unicode — when these tables are absent or corrupt, extractors emit replacement characters, control codes, or seemingly random symbols.

### How accurate is pdf-inspector's garbled text detection?

The system uses **multiple independent heuristics** that must collectively indicate problems before OCR is recommended. The `CipherGarbleStats` module applies statistical tests (letter frequency deviation, shape-cosine similarity) that distinguish genuine low-entropy languages from encoding failures. For critical applications, inspect the `PageTextQualityEvidence` structure to review per-heuristic scores before finalizing OCR routing.

### Can I customize the thresholds for garbage detection?

The current implementation in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) uses **compiled constants** for replacement density thresholds and cipher-likeness scores. For custom sensitivity, wrap the low-level API (`detect_encoding_issues`, `is_garbage_text`) and apply your own logic to the boolean outputs, or fork and modify the `page_replacement_evidence_needs_ocr` threshold parameters.

### Does pdf-inspector perform OCR itself, or only detect when it's needed?

pdf-inspector **detects encoding problems and signals OCR requirements** but does not implement OCR natively. The `OCR_REASON_*` flags and `pages_needing_ocr` / `needs_ocr` fields integrate with external OCR systems — Firecrawl's hosted pipeline uses GPU-accelerated vision models, while self-hosted deployments can route flagged pages to Tesseract, EasyOCR, or cloud APIs based on the structured reason codes.