Handling PDF Encoding Issues and Garbled Text with pdf-inspector: A Complete Guide

The pdf-inspector library automatically detects garbled PDF text using heuristic checks for replacement characters, substitution-cipher patterns, and CID-related garbage, then routes problematic pages to OCR fallback.

PDFs often contain broken font encodings that produce unreadable output — missing ToUnicode CMaps, failed CID-to-Unicode mappings, or glyphs rendered as vector shapes. The firecrawl/pdf-inspector crate solves this through a multi-stage pipeline that isolates encoding problems at both the markdown and individual text-item level, enabling automatic OCR fallback when extraction quality degrades.

How pdf-inspector Detects Encoding Failures

The library operates a detection → extraction → markdown rendering pipeline. During extraction, it processes low-level TextItem structures from the Lopdf parser. When font encoding breaks down, these items contain mangled byte sequences that manifest as replacement characters, control codes, or statistically improbable letter distributions.

pdf-inspector implements two complementary detection layers:

Level	Checks Performed	Source Location
Markdown-level	U+FFFD replacement characters; "dollar-as-space" patterns (Word$Word); substitution-cipher letter statistics; non-alphanumeric dominance	src/text_quality.rs: detect_encoding_issues, is_garbage_text
Item-level	Runs of replacement characters; private-use / C1-control runs; CID-related garbage; statistical substitution-cipher evidence	src/text_quality.rs: analyze_text_quality, region_items_have_decoding_issue

Both layers feed into the same decision logic: when quality checks fail, the page or region receives an OCR reason flag (OCR_REASON_SUSPECTED_GARBLED_TEXT, OCR_REASON_SCANNED, or similar) that downstream code can use to trigger alternative processing.

The Quality Detection Pipeline in src/lib.rs

The public API orchestrates detection through process_pdf_with_options. The flow in src/lib.rs works as follows:

1. process_pdf_with_options loads the PDF and delegates to process_document
2. process_document invokes extract_pages_markdown_mem (or region-specific variants)
3. For each extracted page, the code evaluates quality:

let has_decoding_issue = has_text_quality_issue
    || (!md.is_empty() && (is_cid_garbage(&md) || detect_encoding_issues(&md)));

When has_decoding_issue evaluates to true, the system calls add_ocr_reason to flag the page. The caller receives this signal in the result structure and can initiate OCR fallback.

Text Quality Heuristics in src/text_quality.rs

The text-quality module implements concrete detection algorithms. Each heuristic targets a specific failure mode common in malformed PDFs:

• has_replacement_text_run — Detects contiguous runs of U+FFFD (Unicode replacement character), indicating bytes that could not be decoded to valid Unicode
• has_private_use_text_run / has_cid_control_token — Identifies private-use area codepoints and C1 control characters that leak through when CID font mappings fail
• has_dollar_as_space_pattern — Recognizes the distinctive Word$Word pattern where dollar signs substitute for spaces due to encoding table corruption
• CipherGarbleStats — Performs statistical substitution-cipher detection using letter frequency analysis, shape-cosine similarity, and case-shift bigram modeling
• is_garbage_text — Measures alphanumeric vs. non-alphanumeric ratio while ignoring markdown syntax, flagging outputs dominated by symbols and punctuation

The module aggregates evidence through PageTextQualityEvidence and makes final OCR decisions via page_replacement_evidence_needs_ocr, which weighs replacement character density, run length, and cipher-likeness against configurable thresholds.

Detecting Garbled PDFs Automatically

The simplest integration processes a complete PDF and reports which pages require OCR:

use pdf_inspector::{process_pdf_with_options, PdfOptions, ProcessMode};

fn main() -> Result<(), pdf_inspector::PdfError> {
    let opts = PdfOptions::new()
        .mode(ProcessMode::Full);

    let result = pdf_inspector::process_pdf_with_options("sample.pdf", opts)?;

    println!("PDF type: {:?}", result.pdf_type);
    println!("Pages needing OCR: {:?}", result.pages_needing_ocr);
    
    if let Some(md) = result.markdown {
        println!("Extracted markdown (clean pages only):\n{md}");
    }
    Ok(())
}

The pages_needing_ocr field reveals encoding failures detected during extraction. Use ProcessMode::DetectOnly for faster analysis when you only need quality assessment without full markdown generation.

Region-Level Encoding Detection

For targeted extraction — extracting text from specific coordinates rather than full pages — the library performs quality checks per-region:

use pdf_inspector::extract_text_in_regions_mem;

fn main() -> Result<(), pdf_inspector::PdfError> {
    // Page 0, region from (50,700) to (550,750) in PDF points
    let regions = vec![(0u32, vec![[50.0, 700.0, 550.0, 750.0]])];

    let buffer = std::fs::read("invoice.pdf")?;
    let page_results = extract_text_in_regions_mem(&buffer, &regions)?;

    for page in page_results {
        for (i, region) in page.regions.iter().enumerate() {
            if region.needs_ocr {
                println!("Region {i} on page {} needs OCR", page.page + 1);
            } else {
                println!("Region {i} text: {}", region.text);
            }
        }
    }
    Ok(())
}

Each RegionText exposes a boolean needs_ocr flag populated by the same quality heuristics used for full-page extraction. This enables fine-grained fallback: OCR only the garbled regions while preserving clean extracted text elsewhere.

Using the Low-Level Quality API

For custom pipelines, invoke detection functions directly on markdown strings:

use pdf_inspector::text_quality::{
    detect_encoding_issues, 
    is_garbage_text, 
    is_cid_garbage
};

fn main() {
    let md = "----1-.-.-.___  --.-. .._ I_---.";
    
    println!("Has replacement chars? {}", md.contains('\u{FFFD}'));
    println!("Encoding issues? {}", detect_encoding_issues(md));
    println!("Garbage text? {}", is_garbage_text(md));
    println!("CID garbage? {}", is_cid_garbage(md));
}

This interface supports proactive quality monitoring — validate externally-sourced markdown or compare extraction engines against pdf-inspector's detection criteria.

Integration with OCR Fallback

The encoding detection system produces structured OCR reasons that integrate cleanly with GPU-based OCR pipelines. When add_ocr_reason marks a page with OCR_REASON_SUSPECTED_GARBLED_TEXT, the resulting metadata includes:

• The specific heuristic that triggered (replacement runs, CID garbage, cipher detection)
• Confidence level derived from PageTextQualityEvidence aggregation
• Page or region coordinates for targeted reprocessing

This design separates quality assessment from remediation strategy — your application decides whether to use local OCR, cloud vision APIs, or manual review based on the signaled reason codes.

Key Source Files for PDF Encoding Handling

File	Purpose
src/lib.rs	Public API entry points: process_pdf, detect_pdf, extract_pages_markdown_mem; OCR reason propagation
src/text_quality.rs	All quality heuristics: replacement detection, cipher statistics, garbage classification, evidence aggregation
src/markdown/mod.rs	Markdown generation and layout complexity analysis; integrates quality checks into rendering pipeline
src/extractor/content_stream.rs	Lopdf content-stream parser producing TextItems consumed by quality analysis
src/detector.rs	PDF-type classification including scan detection and OCR triggering

Summary

• pdf-inspector detects PDF encoding issues through markdown-level and item-level heuristics in src/text_quality.rs
• Quality signals include replacement characters (U+FFFD), private-use runs, CID garbage, substitution-cipher statistics, and symbol-to-text ratios
• The process_pdf_with_options API automatically flags pages needing OCR via pages_needing_ocr in the result structure
• Region-level extraction with extract_text_in_regions_mem enables targeted quality assessment for specific document areas
• OCR reasons (OCR_REASON_SUSPECTED_GARBLED_TEXT) provide structured metadata for downstream remediation decisions

Frequently Asked Questions

What causes garbled text in PDF extraction?

PDF font encoding breaks when ToUnicode CMaps are missing, CID-to-Unicode mappings fail, or glyphs are drawn as vector paths rather than encoded text. The binary font data inside PDFs often uses custom encodings that mapping tables should convert to Unicode — when these tables are absent or corrupt, extractors emit replacement characters, control codes, or seemingly random symbols.

How accurate is pdf-inspector's garbled text detection?

The system uses multiple independent heuristics that must collectively indicate problems before OCR is recommended. The CipherGarbleStats module applies statistical tests (letter frequency deviation, shape-cosine similarity) that distinguish genuine low-entropy languages from encoding failures. For critical applications, inspect the PageTextQualityEvidence structure to review per-heuristic scores before finalizing OCR routing.

Can I customize the thresholds for garbage detection?

The current implementation in src/text_quality.rs uses compiled constants for replacement density thresholds and cipher-likeness scores. For custom sensitivity, wrap the low-level API (detect_encoding_issues, is_garbage_text) and apply your own logic to the boolean outputs, or fork and modify the page_replacement_evidence_needs_ocr threshold parameters.

Does pdf-inspector perform OCR itself, or only detect when it's needed?

pdf-inspector detects encoding problems and signals OCR requirements but does not implement OCR natively. The OCR_REASON_* flags and pages_needing_ocr / needs_ocr fields integrate with external OCR systems — Firecrawl's hosted pipeline uses GPU-accelerated vision models, while self-hosted deployments can route flagged pages to Tesseract, EasyOCR, or cloud APIs based on the structured reason codes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}