What Encoding Issues Does pdf-inspector Detect in PDFs?

pdf-inspector detects three critical encoding failure patterns in PDF text extraction: Unicode replacement characters (U+FFFD), dollar-sign-as-space artifacts from broken ToUnicode CMaps, and substitution-cipher-style garbling caused by character offset errors.

The firecrawl/pdf-inspector library validates text extraction quality through statistical heuristics implemented in src/text_quality.rs. When processing documents with corrupted font encodings or invalid character-to-glyph mappings, the detector identifies pages requiring OCR fallback to preserve extraction accuracy.

The Three Core Encoding Issue Heuristics

The primary detection logic resides in the detect_encoding_issues function (lines 31-55 of src/text_quality.rs). This analyzer flags a page when any of the following three heuristics trigger, indicating failed PDF-to-Unicode mapping.

Unicode Replacement Characters (U+FFFD)

The detector scans for the Unicode replacement character \u{FFFD} (U+FFFD) in the extracted Markdown output.

• Technical cause: When the PDF-to-Unicode CMap fails to resolve a byte sequence to a valid character, the extractor inserts the replacement character as a fallback.
• Detection method: A simple presence check in the final text string indicates unmapped glyphs or incomplete font encodings.

Dollar-Sign-as-Space Patterns

This heuristic identifies corrupted ToUnicode CMaps that incorrectly map space characters to dollar signs.

• Pattern: The code searches for $ symbols appearing between letters in Word$Word$Word formations.
• Thresholds: The page is flagged when such patterns occur more than 20 times or when they constitute over 50% of all $ symbols in the document.
• Root cause: Some broken CMaps substitute $ as a surrogate for missing whitespace, creating joined words with intervening dollar signs.

Substitution-Cipher Style Garbling

The most sophisticated detection uses statistical analysis of ASCII letter frequencies via the CipherGarbleStats struct.

• Pattern recognition: Broken CMaps may apply a constant offset to every character (e.g., transforming "Certificate" into "8VceZWZTReV"), creating a perfect alphabet permutation resembling a substitution cipher.
• Dual similarity test:
  • english_cosine < 0.60 indicates low similarity to standard English letter distribution.
  • english_shape_cosine ≥ 0.90 confirms the distribution shape still resembles English, distinguishing cipher garble from random noise or non-English text.

Span-Level Encoding Validation

While detect_encoding_issues operates on complete Markdown pages, complementary functions in src/text_quality.rs analyze individual text spans for granular corruption detection.

Private-Use Area Detection

The has_private_use_text_run function identifies consecutive Private-Use Area (PUA) characters. These often arise from CID-to-Unicode mapping failures where font vendors assign custom glyph IDs that the extractor cannot resolve to standard Unicode code points.

CID Garbage and Control Token Analysis

Two additional helpers detect misinterpreted character identifiers:

• is_cid_garbage: Flags C1-control characters (U+0080–U+009F) or high-Latin-1 characters resulting from CID values incorrectly parsed as Latin-1 bytes.
• has_cid_control_token: Identifies tokens containing unusually high proportions of C1 control bytes.

These span-level checks feed into text_span_decoding_issue_kind, which determines if a single TextItem qualifies as "strongly" garbled, complementing the page-wide heuristics.

Implementation in text_quality.rs

The encoding detection system bridges low-level PDF parsing and high-quality text output through specific source files.

The detect_encoding_issues Function

Located at lines 31-55 of src/text_quality.rs, this function evaluates the three primary heuristics sequentially. When any test returns true, the function immediately flags the page for OCR processing, preventing downstream consumption of corrupted text.

Integration with ToUnicode Handling

The src/tounicode.rs module manages CMap parsing and fallback generation. When this module produces broken mappings—such as constant-offset character translations—the detect_encoding_issues heuristics in text_quality.rs catch these artifacts in the final Markdown output generated by src/markdown/convert.rs.

Detecting Encoding Issues Programmatically

Access encoding detection results through the public API or command-line interface.

Rust API Example

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::options::ProcessOptions;

/// Run the extractor and print whether any page had encoding problems.
fn main() {
    let opts = ProcessOptions::default(); // default opts enable quality checks
    let result = process_pdf_with_options("sample.pdf", opts).unwrap();

    // `has_encoding_issues` is set if any of the heuristics above triggered.
    if result.has_encoding_issues {
        eprintln!("⚠️  Encoding issues detected – OCR may be required.");
    } else {
        println!("✅  No encoding problems found.");
    }
}

Command-Line Usage

pdf2md sample.pdf --json | jq '.has_encoding_issues'

The JSON output contains a boolean has_encoding_issues field directly reflecting the detect_encoding_issues evaluation.

Summary

• pdf-inspector detects encoding failures through three complementary heuristics in src/text_quality.rs: Unicode replacement characters (U+FFFD), dollar-as-space patterns, and substitution-cipher statistical analysis.
• The detect_encoding_issues function (lines 31-55) serves as the primary gatekeeper, triggering OCR fallback when any heuristic fires.
• Span-level validators including has_private_use_text_run and is_cid_garbage provide granular detection of CID-to-Unicode mapping failures.
• Detection results surface through the has_encoding_issues boolean in both the Rust API and JSON CLI output.

Frequently Asked Questions

What triggers the has_encoding_issues flag in pdf-inspector?

The flag triggers when detect_encoding_issues detects any of three patterns: presence of U+FFFD replacement characters indicating unmapped glyphs, excessive $ symbols between words suggesting corrupted space mappings, or statistical evidence of substitution-cipher garbling where letter frequencies are permuted but maintain English distribution shape.

How does pdf-inspector distinguish between normal text and substitution cipher garbling?

The CipherGarbleStats analyzer compares observed ASCII letter frequencies against standard English using cosine similarity. It requires both english_cosine below 0.60 (indicating content divergence from English) and english_shape_cosine at or above 0.90 (confirming the distribution pattern matches English), effectively isolating constant-offset encoding errors from legitimate non-English text.

What are Private-Use Area characters and why do they indicate encoding problems?

Private-Use Area (PUA) characters occupy Unicode code points reserved for vendor-specific glyphs. The has_private_use_text_run function detects long runs of these characters, which typically indicate CID-to-Unicode mapping failures where the extractor cannot resolve custom font glyph IDs to standard Unicode values.

Can pdf-inspector automatically fix encoding issues, or only detect them?

The library focuses exclusively on detection and flagging. When detect_encoding_issues identifies encoding problems, the system sets has_encoding_issues to true, signaling downstream processors to route the page through OCR or alternative extraction methods rather than attempting automated encoding repair.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}