How to Detect Encoding Issues and Garbled Text in PDFs Using pdf‑inspector

pdf‑inspector automatically flags broken font encodings and garbled text through a multi‑stage quality assessment that checks for Unicode replacement characters, garbage‑text ratios, and CID‑font mapping failures.

The pdf‑inspector library (maintained at firecrawl/pdf-inspector) provides built‑in detection for encoding‑corrupted PDFs that would otherwise produce unreadable Markdown output. Rather than silently emitting mangled text, the library surfaces a clear signal you can act on—whether through OCR fallback, manual review, or rejection of the source document.

How pdf‑inspector Detects Encoding Problems

The detection system in pdf‑inspector operates through three complementary checks defined in src/text_quality.rs and integrated into the processing pipeline in src/lib.rs.

Encoding Sanity Check with detect_encoding_issues

The primary screening function, detect_encoding_issues, scans extracted text for hallmark symptoms of broken CMap or font‑encoding tables. According to the source, it identifies:

• Unicode replacement characters (U+FFFD) — inserted when a glyph cannot be mapped to a valid code point
• Dollar‑sign placeholders ($) — commonly used by PDF generators as fallback for unknown glyphs
• Financial symbols (¤, §) — frequently leaked by corrupted encodings masquerading as spaces or punctuation

This function resides at line 40 of src/text_quality.rs and serves as the first line of defense against subtle encoding corruption.

Garbage‑Text Ratio Heuristic

When encoding failures produce visually scrambled output, the is_garbage_text function catches them through statistical analysis. The implementation (lines 9–31 of src/text_quality.rs) computes the ratio of alphanumeric characters to total characters. Text falling below a safe threshold—such as strings like ----1-.-.-.___ --.-.—is flagged as unreadable garbage regardless of specific character patterns.

CID‑Font Fallback Detection

PDFs using CID‑encoded fonts without proper ToUnicode maps produce fundamentally undecodable text. The helper is_cid_garbage detects these cases and forces OCR processing. This check is wired into the quality assessment pipeline at lines 601–607 of src/lib.rs, ensuring CID‑garbage pages never pass through as "successful" extraction.

Accessing Detection Results

All three signals feed into a single boolean field exposed through pdf‑inspector's public API.

Rust Library Usage

When using process_pdf_with_options, check the has_encoding_issues field on the returned PDFProcessingResult:

use pdf_inspector::{process_pdf_with_options, ProcessOptions};

fn main() {
    let pdf_path = "document.pdf";
    let opts = ProcessOptions::default();
    let result = process_pdf_with_options(pdf_path, opts);

    println!("Extracted Markdown:\n{}", result.markdown);
    
    if result.has_encoding_issues {
        eprintln!("⚠️  Encoding issues detected — OCR was applied to affected pages.");
    }
}

The has_encoding_issues field aggregates findings from all three detection stages, giving you a single reliable signal.

CLI Usage with --json Flag

The pdf2md binary exposes the same information through JSON output:

# Extract with full metadata

pdf2md report.pdf --json > output.json

# Query the encoding flag directly

jq '.has_encoding_issues' output.json

# Conditional processing in shell scripts

if pdf2md report.pdf --json | jq -e '.has_encoding_issues' > /dev/null; then
    echo "Document requires OCR review"
fi

When has_encoding_issues is true, pdf‑inspector automatically adds affected pages to the OCR processing list, ensuring final output contains legible text.

Stand‑alone Quality Check

For applications that need to validate already‑extracted strings, import the quality module directly:

use pdf_inspector::text_quality::detect_encoding_issues;

fn main() {
    let candidate_text = "Price: $ 100 ¤";
    
    if detect_encoding_issues(&candidate_text) {
        println!("Encoding corruption detected — source PDF needs review.");
    }
}

Core Implementation Files

File	Role	Key Functions
src/text_quality.rs	Detection algorithms	detect_encoding_issues, is_garbage_text, is_cid_garbage
src/lib.rs	Public API integration	process_pdf_with_options, quality flag aggregation at lines 601–607
src/detector.rs	Document type classification	PDF type determination, OCR fallback orchestration
src/bin/pdf2md.rs	CLI frontend	--json flag implementation, metadata serialization

These components work together to ensure encoding problems are caught early and handled transparently, whether you're batch‑processing documents or building custom extraction pipelines.

Summary

• Three detection layers — detect_encoding_issues for specific corruption patterns, is_garbage_text for statistical garbage detection, and is_cid_garbage for CID‑font mapping failures
• Single boolean result — has_encoding_issues in PDFProcessingResult consolidates all quality signals
• Automatic OCR fallback — flagged pages are routed to OCR without manual intervention
• CLI and API parity — same detection logic available via pdf2md --json or direct Rust calls
• Source locations — core logic in src/text_quality.rs (lines 9–40), integration in src/lib.rs (lines 601–607)

Frequently Asked Questions

How accurate is pdf‑inspector's encoding detection?

The detection achieves high precision by combining pattern‑matching for known corruption signatures with statistical validation. The is_garbage_text ratio check minimizes false positives from legitimate symbol‑heavy content, while detect_encoding_issues catches subtle encoding failures that pure statistics might miss.

Can I customize the detection thresholds?

Currently, pdf‑inspector uses fixed thresholds tuned for general‑purpose document processing. The library does not expose threshold parameters in ProcessOptions. For specialized domains, you can post‑process the markdown output using the stand‑alone detect_encoding_issues function with custom logic.

Does encoding detection work for all PDF types?

Detection applies to text‑based and mixed PDFs where font encoding information is present but potentially corrupt. Purely scanned PDFs bypass encoding detection entirely and proceed directly to OCR, as determined by the classifier in src/detector.rs.

What happens when encoding issues are detected?

pdf‑inspector automatically appends affected pages to the OCR queue, then merges OCR results with any successfully extracted text. The final markdown output contains fully legible content, and has_encoding_issues remains true to indicate that OCR was required.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}