# How to Detect Encoding Issues and Garbled Text in PDFs Using pdf‑inspector

> Easily detect PDF encoding issues and garbled text with pdf-inspector. Our tool analyzes font encodings, Unicode replacement characters, and CID-font mapping for accurate results.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: how-to-guide
- Published: 2026-08-11

---

**pdf‑inspector automatically flags broken font encodings and garbled text through a multi‑stage quality assessment that checks for Unicode replacement characters, garbage‑text ratios, and CID‑font mapping failures.**

The pdf‑inspector library (maintained at `firecrawl/pdf-inspector`) provides built‑in detection for encoding‑corrupted PDFs that would otherwise produce unreadable Markdown output. Rather than silently emitting mangled text, the library surfaces a clear signal you can act on—whether through OCR fallback, manual review, or rejection of the source document.

## How pdf‑inspector Detects Encoding Problems

The detection system in pdf‑inspector operates through three complementary checks defined in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) and integrated into the processing pipeline in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs).

### Encoding Sanity Check with `detect_encoding_issues`

The primary screening function, `detect_encoding_issues`, scans extracted text for hallmark symptoms of broken CMap or font‑encoding tables. According to the source, it identifies:

- **Unicode replacement characters (`U+FFFD`)** — inserted when a glyph cannot be mapped to a valid code point
- **Dollar‑sign placeholders (`$`)** — commonly used by PDF generators as fallback for unknown glyphs
- **Financial symbols (`¤`, `§`)** — frequently leaked by corrupted encodings masquerading as spaces or punctuation

This function resides at line 40 of [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) and serves as the first line of defense against subtle encoding corruption.

### Garbage‑Text Ratio Heuristic

When encoding failures produce visually scrambled output, the `is_garbage_text` function catches them through statistical analysis. The implementation (lines 9–31 of [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs)) computes the ratio of alphanumeric characters to total characters. Text falling below a safe threshold—such as strings like `----1-.-.-.___ --.-.`—is flagged as unreadable garbage regardless of specific character patterns.

### CID‑Font Fallback Detection

PDFs using **CID‑encoded fonts** without proper ToUnicode maps produce fundamentally undecodable text. The helper `is_cid_garbage` detects these cases and forces OCR processing. This check is wired into the quality assessment pipeline at lines 601–607 of [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs), ensuring CID‑garbage pages never pass through as "successful" extraction.

## Accessing Detection Results

All three signals feed into a single boolean field exposed through pdf‑inspector's public API.

### Rust Library Usage

When using `process_pdf_with_options`, check the `has_encoding_issues` field on the returned `PDFProcessingResult`:

```rust
use pdf_inspector::{process_pdf_with_options, ProcessOptions};

fn main() {
    let pdf_path = "document.pdf";
    let opts = ProcessOptions::default();
    let result = process_pdf_with_options(pdf_path, opts);

    println!("Extracted Markdown:\n{}", result.markdown);
    
    if result.has_encoding_issues {
        eprintln!("⚠️  Encoding issues detected — OCR was applied to affected pages.");
    }
}

```

The `has_encoding_issues` field aggregates findings from all three detection stages, giving you a single reliable signal.

### CLI Usage with `--json` Flag

The `pdf2md` binary exposes the same information through JSON output:

```bash

# Extract with full metadata

pdf2md report.pdf --json > output.json

# Query the encoding flag directly

jq '.has_encoding_issues' output.json

# Conditional processing in shell scripts

if pdf2md report.pdf --json | jq -e '.has_encoding_issues' > /dev/null; then
    echo "Document requires OCR review"
fi

```

When `has_encoding_issues` is `true`, pdf‑inspector automatically adds affected pages to the OCR processing list, ensuring final output contains legible text.

### Stand‑alone Quality Check

For applications that need to validate already‑extracted strings, import the quality module directly:

```rust
use pdf_inspector::text_quality::detect_encoding_issues;

fn main() {
    let candidate_text = "Price: $ 100 ¤";
    
    if detect_encoding_issues(&candidate_text) {
        println!("Encoding corruption detected — source PDF needs review.");
    }
}

```

## Core Implementation Files

| File | Role | Key Functions |
|------|------|---------------|
| [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) | Detection algorithms | `detect_encoding_issues`, `is_garbage_text`, `is_cid_garbage` |
| [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) | Public API integration | `process_pdf_with_options`, quality flag aggregation at lines 601–607 |
| [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) | Document type classification | PDF type determination, OCR fallback orchestration |
| [`src/bin/pdf2md.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/bin/pdf2md.rs) | CLI frontend | `--json` flag implementation, metadata serialization |

These components work together to ensure encoding problems are caught early and handled transparently, whether you're batch‑processing documents or building custom extraction pipelines.

## Summary

- **Three detection layers** — `detect_encoding_issues` for specific corruption patterns, `is_garbage_text` for statistical garbage detection, and `is_cid_garbage` for CID‑font mapping failures
- **Single boolean result** — `has_encoding_issues` in `PDFProcessingResult` consolidates all quality signals
- **Automatic OCR fallback** — flagged pages are routed to OCR without manual intervention
- **CLI and API parity** — same detection logic available via `pdf2md --json` or direct Rust calls
- **Source locations** — core logic in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) (lines 9–40), integration in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) (lines 601–607)

## Frequently Asked Questions

### How accurate is pdf‑inspector's encoding detection?

The detection achieves high precision by combining pattern‑matching for known corruption signatures with statistical validation. The `is_garbage_text` ratio check minimizes false positives from legitimate symbol‑heavy content, while `detect_encoding_issues` catches subtle encoding failures that pure statistics might miss.

### Can I customize the detection thresholds?

Currently, pdf‑inspector uses fixed thresholds tuned for general‑purpose document processing. The library does not expose threshold parameters in `ProcessOptions`. For specialized domains, you can post‑process the `markdown` output using the stand‑alone `detect_encoding_issues` function with custom logic.

### Does encoding detection work for all PDF types?

Detection applies to text‑based and mixed PDFs where font encoding information is present but potentially corrupt. Purely scanned PDFs bypass encoding detection entirely and proceed directly to OCR, as determined by the classifier in [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs).

### What happens when encoding issues are detected?

pdf‑inspector automatically appends affected pages to the OCR queue, then merges OCR results with any successfully extracted text. The final `markdown` output contains fully legible content, and `has_encoding_issues` remains `true` to indicate that OCR was required.