How pdf-inspector's Garbage Text Detection Upgrades Mixed PDFs to Scanned Status

pdf-inspector uses character-ratio heuristics and CID-font pattern matching to flag garbage text in Mixed PDFs, then upgrades them to Scanned status so downstream pipelines trigger OCR instead of using corrupted vector-text layers.

The pdf-inspector library from Firecrawl classifies documents into three coarse types—TextBased, Mixed, and Scanned—based on the quality of extracted text. When a document initially tests as Mixed, meaning it contains both vector text and raster images, the engine runs an additional garbage-text detection step on the Markdown produced from vector-text extraction. This safeguard prevents mojibake and unreadable character sequences from reaching downstream consumers.

Garbage-Text Detection Heuristics

The core quality checks live in src/text_quality.rs. Two complementary functions evaluate whether extracted text is usable or corrupted.

Character Ratio Analysis with is_garbage_text

The is_garbage_text function scores text by comparing alphanumeric to non-alphanumeric characters.

// In src/text_quality.rs, lines 31-71
// Pseudocode representation of the logic:
fn is_garbage_text(text: &str) -> bool {
    let total = text.chars().count();
    if total < 50 {
        return false; // Too short to judge
    }
    
    let alphanumeric = text.chars()
        .filter(|c| c.is_alphanumeric())
        .count();
    
    // If less than half are alphanumeric → garbage
    (alphanumeric as f64 / total as f64) < 0.5
}

The function ignores typical Markdown syntax and decorative leaders during counting. This prevents false positives from documents that legitimately contain formatting characters.

CID Font Failure Detection with is_cid_garbage

A secondary check, is_cid_garbage, addresses a specific failure mode: CID fonts without valid ToUnicode mappings. These fonts encode glyphs as character IDs rather than Unicode codepoints, producing unreadable output when extracted naively.

// In src/text_quality.rs, lines 73-119
// Looks for:
// - Excess C1 control characters (0x80-0x9F)
// - High-Latin-1 mojibake patterns
// - Other CID-specific corruption signatures
fn is_cid_garbage(text: &str) -> bool {
    if !is_garbage_text(text) {
        return false;
    }
    // Additional CID-specific heuristics...
}

This function calls is_garbage_text first, then applies stricter pattern matching. Together, these heuristics catch the two dominant causes of unreadable vector text: general character corruption and font encoding failures.

The Mixed-to-Scanned Upgrade Logic

After the extraction pipeline finishes, src/lib.rs evaluates whether a Mixed PDF should be reclassified. The decision point occurs at lines 443-450.

Upgrade Conditions

The engine upgrades a Mixed PDF to Scanned status when all of the following hold:

• PdfType::Mixed was detected during initial classification
• Vector-text Markdown was successfully generated (Option<String> is Some)
• is_garbage_text returns true on that Markdown

When triggered, the upgrade:

1. Discards the garbage Markdown (sets it to None)
2. Reclassifies the document as PdfType::Scanned
3. Assigns confidence 0.95 that the text layer is unreliable

// In src/lib.rs, lines 443-450
// Upgrade logic (adapted from source):
if pdf_type == PdfType::Mixed && markdown.is_some() {
    if is_garbage_text(markdown.as_ref().unwrap()) {
        // Upgrade to Scanned — discard garbage text
        pdf_type = PdfType::Scanned;
        markdown = None;
        confidence = 0.95;
    }
}

This signals downstream consumers to ignore the vector-text extraction and execute a full OCR pass instead.

Practical Usage Examples

Rust API

The upgrade happens transparently when using process_pdf_with_options:

use pdf_inspector::{process_pdf_with_options, PdfType, ProcessOptions};

let opts = ProcessOptions::default();
let (md, layout, enc, _, _, _) = 
    process_pdf_with_options("sample.pdf", opts);

// md is Option<String> — None if upgraded to Scanned
match layout.pdf_type() {
    PdfType::Scanned => {
        println!("Mixed PDF upgraded to Scanned – OCR required");
        // md is None here; fetch OCR text separately
    },
    PdfType::Mixed => {
        println!("Mixed PDF retained – using vector text");
        let text = md.unwrap();
    },
    PdfType::TextBased => {
        println!("Pure text PDF – no OCR needed");
    },
}

Command-Line Tool

The same logic applies when using the bundled pdf2md utility:

# Automatically handles Mixed→Scanned upgrades

pdf2md sample.pdf

# With verbose output to see reclassification

pdf2md --verbose corrupted-font.pdf

# Output: "Detected Mixed PDF, garbage text found, upgrading to Scanned"

Why Mixed PDFs Need This Safeguard

Mixed PDFs are particularly susceptible to hidden text-layer corruption. Common scenarios include:

• Identity-H fonts embedded without ToUnicode maps
• Legacy CID fonts from pre-Unicode publishing workflows
• Scanned documents with "invisible" OCR text layers that contain positioning artifacts

Without garbage detection, these documents would return nonsensical mojibake that appears structurally valid but is semantically useless. The upgrade to Scanned ensures clients receive either clean vector text or a clear signal that OCR is required—never corrupted intermediate output.

Summary

• Garbage detection runs in src/text_quality.rs via is_garbage_text and is_cid_garbage
• Character-ratio analysis flags text where <50% of characters ≥50 total are alphanumeric
• CID-specific checks catch font encoding failures that pure ratio analysis misses
• Upgrade logic in src/lib.rs lines 443-450 reclassifies Mixed→Scanned when garbage is confirmed
• Result: Markdown is discarded, confidence set to 0.95, and downstream systems route to OCR

Frequently Asked Questions

What threshold triggers garbage detection in pdf-inspector?

The is_garbage_text function requires at least 50 total characters before judging quality, then flags text as garbage when less than 50% are alphanumeric. This filters out short snippets and decorative content while catching substantial corruption.

Why upgrade Mixed PDFs to Scanned instead of TextBased?

Scanned status preserves the document's true structure: it contains raster images that need OCR, plus an unreliable text layer. Upgrading to TextBased would incorrectly promise usable vector text. The Scanned classification ensures downstream pipelines invoke full OCR rather than trusting corrupted extractions.

Which font problems does is_cid_garbage specifically target?

The function detects CID font failures including Identity-H encoding without ToUnicode maps, excess C1 control characters, and high-Latin-1 mojibake patterns. These occur when glyph IDs cannot map to Unicode codepoints, producing sequences like "ˇ˛˝˚" instead of readable text.

Can I disable the Mixed-to-Scanned upgrade behavior?

pdf-inspector does not expose a direct toggle for this safeguard. The upgrade is integral to quality guarantees in src/lib.rs. To bypass it, you would need to fork and modify the post-extraction logic at lines 443-450, though this risks propagating garbage text to production systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}