# What Encoding Issues Does pdf-inspector Detect in PDFs?

> pdf-inspector finds three key PDF text extraction encoding issues: Unicode replacement characters, dollar-sign-as-space artifacts, and garbled text from character offset errors. Improve your PDF data accuracy.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-08-13

---

**pdf-inspector detects three critical encoding failure patterns in PDF text extraction: Unicode replacement characters (U+FFFD), dollar-sign-as-space artifacts from broken ToUnicode CMaps, and substitution-cipher-style garbling caused by character offset errors.**

The firecrawl/pdf-inspector library validates text extraction quality through statistical heuristics implemented in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs). When processing documents with corrupted font encodings or invalid character-to-glyph mappings, the detector identifies pages requiring OCR fallback to preserve extraction accuracy.

## The Three Core Encoding Issue Heuristics

The primary detection logic resides in the `detect_encoding_issues` function (lines 31-55 of [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs)). This analyzer flags a page when **any** of the following three heuristics trigger, indicating failed PDF-to-Unicode mapping.

### Unicode Replacement Characters (U+FFFD)

The detector scans for the Unicode replacement character `\u{FFFD}` (U+FFFD) in the extracted Markdown output.

- **Technical cause**: When the PDF-to-Unicode CMap fails to resolve a byte sequence to a valid character, the extractor inserts the replacement character as a fallback.
- **Detection method**: A simple presence check in the final text string indicates unmapped glyphs or incomplete font encodings.

### Dollar-Sign-as-Space Patterns

This heuristic identifies corrupted ToUnicode CMaps that incorrectly map space characters to dollar signs.

- **Pattern**: The code searches for `$` symbols appearing between letters in `Word$Word$Word` formations.
- **Thresholds**: The page is flagged when such patterns occur more than 20 times **or** when they constitute over 50% of all `$` symbols in the document.
- **Root cause**: Some broken CMaps substitute `$` as a surrogate for missing whitespace, creating joined words with intervening dollar signs.

### Substitution-Cipher Style Garbling

The most sophisticated detection uses statistical analysis of ASCII letter frequencies via the `CipherGarbleStats` struct.

- **Pattern recognition**: Broken CMaps may apply a constant offset to every character (e.g., transforming "Certificate" into "8VceZWZTReV"), creating a perfect alphabet permutation resembling a substitution cipher.
- **Dual similarity test**:
  - `english_cosine` < 0.60 indicates low similarity to standard English letter distribution.
  - `english_shape_cosine` ≥ 0.90 confirms the distribution shape still resembles English, distinguishing cipher garble from random noise or non-English text.

## Span-Level Encoding Validation

While `detect_encoding_issues` operates on complete Markdown pages, complementary functions in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) analyze individual text spans for granular corruption detection.

### Private-Use Area Detection

The `has_private_use_text_run` function identifies consecutive Private-Use Area (PUA) characters. These often arise from CID-to-Unicode mapping failures where font vendors assign custom glyph IDs that the extractor cannot resolve to standard Unicode code points.

### CID Garbage and Control Token Analysis

Two additional helpers detect misinterpreted character identifiers:

- **`is_cid_garbage`**: Flags C1-control characters (U+0080–U+009F) or high-Latin-1 characters resulting from CID values incorrectly parsed as Latin-1 bytes.
- **`has_cid_control_token`**: Identifies tokens containing unusually high proportions of C1 control bytes.

These span-level checks feed into `text_span_decoding_issue_kind`, which determines if a single `TextItem` qualifies as "strongly" garbled, complementing the page-wide heuristics.

## Implementation in text_quality.rs

The encoding detection system bridges low-level PDF parsing and high-quality text output through specific source files.

### The detect_encoding_issues Function

Located at lines 31-55 of [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs), this function evaluates the three primary heuristics sequentially. When **any** test returns `true`, the function immediately flags the page for OCR processing, preventing downstream consumption of corrupted text.

### Integration with ToUnicode Handling

The [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) module manages CMap parsing and fallback generation. When this module produces broken mappings—such as constant-offset character translations—the `detect_encoding_issues` heuristics in [`text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/text_quality.rs) catch these artifacts in the final Markdown output generated by [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs).

## Detecting Encoding Issues Programmatically

Access encoding detection results through the public API or command-line interface.

### Rust API Example

```rust
use pdf_inspector::process_pdf_with_options;
use pdf_inspector::options::ProcessOptions;

/// Run the extractor and print whether any page had encoding problems.
fn main() {
    let opts = ProcessOptions::default(); // default opts enable quality checks
    let result = process_pdf_with_options("sample.pdf", opts).unwrap();

    // `has_encoding_issues` is set if any of the heuristics above triggered.
    if result.has_encoding_issues {
        eprintln!("⚠️  Encoding issues detected – OCR may be required.");
    } else {
        println!("✅  No encoding problems found.");
    }
}

```

### Command-Line Usage

```bash
pdf2md sample.pdf --json | jq '.has_encoding_issues'

```

The JSON output contains a boolean `has_encoding_issues` field directly reflecting the `detect_encoding_issues` evaluation.

## Summary

- **pdf-inspector** detects encoding failures through three complementary heuristics in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs): Unicode replacement characters (U+FFFD), dollar-as-space patterns, and substitution-cipher statistical analysis.
- The `detect_encoding_issues` function (lines 31-55) serves as the primary gatekeeper, triggering OCR fallback when any heuristic fires.
- Span-level validators including `has_private_use_text_run` and `is_cid_garbage` provide granular detection of CID-to-Unicode mapping failures.
- Detection results surface through the `has_encoding_issues` boolean in both the Rust API and JSON CLI output.

## Frequently Asked Questions

### What triggers the `has_encoding_issues` flag in pdf-inspector?

The flag triggers when `detect_encoding_issues` detects any of three patterns: presence of U+FFFD replacement characters indicating unmapped glyphs, excessive `$` symbols between words suggesting corrupted space mappings, or statistical evidence of substitution-cipher garbling where letter frequencies are permuted but maintain English distribution shape.

### How does pdf-inspector distinguish between normal text and substitution cipher garbling?

The `CipherGarbleStats` analyzer compares observed ASCII letter frequencies against standard English using cosine similarity. It requires both `english_cosine` below 0.60 (indicating content divergence from English) **and** `english_shape_cosine` at or above 0.90 (confirming the distribution pattern matches English), effectively isolating constant-offset encoding errors from legitimate non-English text.

### What are Private-Use Area characters and why do they indicate encoding problems?

Private-Use Area (PUA) characters occupy Unicode code points reserved for vendor-specific glyphs. The `has_private_use_text_run` function detects long runs of these characters, which typically indicate CID-to-Unicode mapping failures where the extractor cannot resolve custom font glyph IDs to standard Unicode values.

### Can pdf-inspector automatically fix encoding issues, or only detect them?

The library focuses exclusively on detection and flagging. When `detect_encoding_issues` identifies encoding problems, the system sets `has_encoding_issues` to `true`, signaling downstream processors to route the page through OCR or alternative extraction methods rather than attempting automated encoding repair.