# How pdf-inspector's Garbage Text Detection Upgrades Mixed PDFs to Scanned Status

> Discover how pdf-inspector's garbage text detection upgrades Mixed PDFs to Scanned status by identifying and flagging corrupted text layers, ensuring accurate OCR processing.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: deep-dive
- Published: 2026-09-02

---

**pdf-inspector uses character-ratio heuristics and CID-font pattern matching to flag garbage text in Mixed PDFs, then upgrades them to Scanned status so downstream pipelines trigger OCR instead of using corrupted vector-text layers.**

The **pdf-inspector** library from Firecrawl classifies documents into three coarse types—**TextBased**, **Mixed**, and **Scanned**—based on the quality of extracted text. When a document initially tests as *Mixed*, meaning it contains both vector text and raster images, the engine runs an additional garbage-text detection step on the Markdown produced from vector-text extraction. This safeguard prevents mojibake and unreadable character sequences from reaching downstream consumers.

## Garbage-Text Detection Heuristics

The core quality checks live in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs). Two complementary functions evaluate whether extracted text is usable or corrupted.

### Character Ratio Analysis with `is_garbage_text`

The `is_garbage_text` function scores text by comparing alphanumeric to non-alphanumeric characters.

```rust
// In src/text_quality.rs, lines 31-71
// Pseudocode representation of the logic:
fn is_garbage_text(text: &str) -> bool {
    let total = text.chars().count();
    if total < 50 {
        return false; // Too short to judge
    }
    
    let alphanumeric = text.chars()
        .filter(|c| c.is_alphanumeric())
        .count();
    
    // If less than half are alphanumeric → garbage
    (alphanumeric as f64 / total as f64) < 0.5
}

```

The function ignores typical Markdown syntax and decorative leaders during counting. This prevents false positives from documents that legitimately contain formatting characters.

### CID Font Failure Detection with `is_cid_garbage`

A secondary check, `is_cid_garbage`, addresses a specific failure mode: **CID fonts without valid ToUnicode mappings**. These fonts encode glyphs as character IDs rather than Unicode codepoints, producing unreadable output when extracted naively.

```rust
// In src/text_quality.rs, lines 73-119
// Looks for:
// - Excess C1 control characters (0x80-0x9F)
// - High-Latin-1 mojibake patterns
// - Other CID-specific corruption signatures
fn is_cid_garbage(text: &str) -> bool {
    if !is_garbage_text(text) {
        return false;
    }
    // Additional CID-specific heuristics...
}

```

This function calls `is_garbage_text` first, then applies stricter pattern matching. Together, these heuristics catch the two dominant causes of unreadable vector text: **general character corruption** and **font encoding failures**.

## The Mixed-to-Scanned Upgrade Logic

After the extraction pipeline finishes, [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) evaluates whether a Mixed PDF should be reclassified. The decision point occurs at lines 443-450.

### Upgrade Conditions

The engine upgrades a Mixed PDF to Scanned status when **all** of the following hold:

- `PdfType::Mixed` was detected during initial classification
- Vector-text Markdown was successfully generated (`Option<String>` is `Some`)
- `is_garbage_text` returns `true` on that Markdown

When triggered, the upgrade:

1. **Discards the garbage Markdown** (sets it to `None`)
2. **Reclassifies the document** as `PdfType::Scanned`
3. **Assigns confidence 0.95** that the text layer is unreliable

```rust
// In src/lib.rs, lines 443-450
// Upgrade logic (adapted from source):
if pdf_type == PdfType::Mixed && markdown.is_some() {
    if is_garbage_text(markdown.as_ref().unwrap()) {
        // Upgrade to Scanned — discard garbage text
        pdf_type = PdfType::Scanned;
        markdown = None;
        confidence = 0.95;
    }
}

```

This signals downstream consumers to **ignore the vector-text extraction** and execute a full OCR pass instead.

## Practical Usage Examples

### Rust API

The upgrade happens transparently when using `process_pdf_with_options`:

```rust
use pdf_inspector::{process_pdf_with_options, PdfType, ProcessOptions};

let opts = ProcessOptions::default();
let (md, layout, enc, _, _, _) = 
    process_pdf_with_options("sample.pdf", opts);

// md is Option<String> — None if upgraded to Scanned
match layout.pdf_type() {
    PdfType::Scanned => {
        println!("Mixed PDF upgraded to Scanned – OCR required");
        // md is None here; fetch OCR text separately
    },
    PdfType::Mixed => {
        println!("Mixed PDF retained – using vector text");
        let text = md.unwrap();
    },
    PdfType::TextBased => {
        println!("Pure text PDF – no OCR needed");
    },
}

```

### Command-Line Tool

The same logic applies when using the bundled `pdf2md` utility:

```bash

# Automatically handles Mixed→Scanned upgrades

pdf2md sample.pdf

# With verbose output to see reclassification

pdf2md --verbose corrupted-font.pdf

# Output: "Detected Mixed PDF, garbage text found, upgrading to Scanned"

```

## Why Mixed PDFs Need This Safeguard

Mixed PDFs are particularly susceptible to hidden text-layer corruption. Common scenarios include:

- **Identity-H fonts** embedded without ToUnicode maps
- **Legacy CID fonts** from pre-Unicode publishing workflows
- **Scanned documents with "invisible" OCR text layers** that contain positioning artifacts

Without garbage detection, these documents would return **nonsensical mojibake** that appears structurally valid but is semantically useless. The upgrade to Scanned ensures clients receive either **clean vector text** or a clear signal that **OCR is required**—never corrupted intermediate output.

## Summary

- **Garbage detection** runs in [`src/text_quality.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_quality.rs) via `is_garbage_text` and `is_cid_garbage`
- **Character-ratio analysis** flags text where <50% of characters ≥50 total are alphanumeric
- **CID-specific checks** catch font encoding failures that pure ratio analysis misses
- **Upgrade logic** in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) lines 443-450 reclassifies Mixed→Scanned when garbage is confirmed
- **Result**: Markdown is discarded, confidence set to 0.95, and downstream systems route to OCR

## Frequently Asked Questions

### What threshold triggers garbage detection in pdf-inspector?

The `is_garbage_text` function requires **at least 50 total characters** before judging quality, then flags text as garbage when **less than 50% are alphanumeric**. This filters out short snippets and decorative content while catching substantial corruption.

### Why upgrade Mixed PDFs to Scanned instead of TextBased?

**Scanned status** preserves the document's true structure: it contains raster images that need OCR, plus an unreliable text layer. Upgrading to **TextBased** would incorrectly promise usable vector text. The Scanned classification ensures downstream pipelines invoke full OCR rather than trusting corrupted extractions.

### Which font problems does `is_cid_garbage` specifically target?

The function detects **CID font failures** including Identity-H encoding without ToUnicode maps, excess C1 control characters, and high-Latin-1 mojibake patterns. These occur when glyph IDs cannot map to Unicode codepoints, producing sequences like "ˇ˛˝˚" instead of readable text.

### Can I disable the Mixed-to-Scanned upgrade behavior?

pdf-inspector does not expose a direct toggle for this safeguard. The upgrade is **integral to quality guarantees** in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs). To bypass it, you would need to fork and modify the post-extraction logic at lines 443-450, though this risks propagating garbage text to production systems.