Hybrid OCR Pipelines in pdf‑inspector: Region‑Based Text Extraction and Layout Models Explained

pdf‑inspector implements hybrid OCR by first classifying PDFs to identify which pages need OCR, then extracting positioned text once for the entire document, and finally running GPU‑accelerated OCR only on the specific problematic regions.

The firecrawl/pdf-inspector Rust crate solves the expensive problem of OCR‑ing entire PDFs by splitting processing into lightweight classification, region‑based extraction, and targeted OCR fallback. This architecture minimizes GPU usage while maintaining accuracy across scanned documents, mixed PDFs, and natively text‑based files.

Lightweight PDF Classification Stage

The hybrid pipeline starts with fast PDF classification that makes OCR decisions without heavy processing. The classify_pdf_mem function in src/lib.rs (lines 89‑101) peeks at the PDF structure to determine its type and flag pages requiring OCR:

let classification = pdf_inspector::classify_pdf_mem(&pdf_bytes)?;
println!("Pages needing OCR: {:?}", classification.pages_needing_ocr);

This function returns a PdfClassification struct defined at src/lib.rs lines 73‑87, containing:

• pdf_type – detected category (TextBased, Scanned, or Mixed)
• pages_needing_ocr – zero‑indexed list of pages requiring fallback OCR
• total_pages and confidence score for the classification

The classification stage completes in approximately 10‑50 milliseconds, making the OCR decision cheap enough to run on every document.

Region‑Based Text Extraction Architecture

pdf‑inspector implements region‑based text extraction (explicitly noted in the comment at src/lib.rs lines 72‑76) to avoid redundant processing. The approach operates in two distinct phases:

Full‑Document Geometry Extraction

The extractor reads the PDF once and builds complete geometric data through extract_positioned_text_from_doc. This captures:

• Positioned text items with precise coordinates
• Bounding rectangles for text regions
• Line geometry and spatial relationships

This single‑pass extraction means layout analysis operates on in‑memory data with near‑zero incremental cost.

Per‑Page Slicing with OCR Flags

The extract_pages_markdown_mem function slices the pre‑computed geometry per page. Each PageMarkdown struct (defined at src/lib.rs lines 9‑20) includes:

pub struct PageMarkdown {
    pub page: usize,
    pub markdown: String,
    pub needs_ocr: bool,  // ← Key flag for hybrid pipeline
    // ... layout metadata
}

The needs_ocr flag triggers when native text is unreliable—specifically for GID‑encoded fonts, broken encodings, or garbled vector text. This region‑based approach ensures only problematic pages proceed to OCR, not entire documents.

Layout Model Integration

pdf‑inspector embeds cheap layout detectors that run during geometric extraction in src/extractor/mod.rs. These detectors analyze the positioned text and rectangles already held in memory:

• Table detection – identifies tabular structures via rectangle‑based, line‑based, and heuristic strategies in src/tables/
• Multi‑column layout detection – recognizes complex reading orders
• Spatial analysis – uses line geometry for structural understanding

Results populate the PagesExtractionResult struct at src/lib.rs lines 22‑38, which carries tables, column information, and OCR flags through the pipeline. Because these models operate on extracted geometry rather than rasterized images, they add negligible overhead.

The Complete Hybrid Pipeline Flow

The three‑stage hybrid OCR workflow orchestrates native extraction and GPU fallback:

1. Classify – Call classify_pdf_mem or process_pdf_with_options to identify Scanned pages and specific pages_needing_ocr
2. Extract with metadata – Run extract_pages_markdown_mem; pages with needs_ocr == false yield reliable markdown immediately
3. Targeted OCR – Send only flagged page images to GPU engines like PaddleOCR or Tesseract, then merge results

use pdf_inspector::{classify_pdf_mem, extract_pages_markdown_mem};

fn process_hybrid(pdf_bytes: &[u8]) -> Result<(), pdf_inspector::PdfError> {
    // Stage 1: Lightweight classification
    let classification = classify_pdf_mem(pdf_bytes)?;
    
    // Stage 2: Region-based extraction
    let extraction = extract_pages_markdown_mem(pdf_bytes, None)?;
    
    for page in extraction.pages {
        if page.needs_ocr {
            // Stage 3: GPU OCR for this specific region only
            let ocr_text = run_gpu_ocr_on_page(page.page);
            merge_into_output(ocr_text);
        } else {
            // Native extraction is reliable — use directly
            output_markdown(page.markdown);
        }
    }
    Ok(())
}

This region‑based design avoids sending whole PDFs to OCR engines, dramatically reducing GPU load and processing time.

Performance Characteristics

The hybrid architecture achieves efficiency through three key properties:

• Single extraction pass – Full‑document geometry computation happens once regardless of page count
• In‑memory layout analysis – Table and column detectors reuse positioned data without re‑parsing
• Surgical OCR invocation – GPU engines process only regions where native text fails

The result preserves native text quality for clean, properly encoded PDFs while maintaining scanned document support through targeted fallback.

Summary

• classify_pdf_mem in src/lib.rs performs fast PDF classification (10‑50ms) to identify pages needing OCR
• Region‑based extraction builds full‑document geometry once, then slices per‑page with needs_ocr flags
• Layout models for tables and columns run cheaply on in‑memory geometric data in src/extractor/mod.rs and src/tables/
• Targeted OCR sends only problematic regions to GPU engines, avoiding whole‑document processing
• The PageMarkdown and PagesExtractionResult structs carry extraction metadata through the pipeline

Frequently Asked Questions

How does pdf‑inspector decide which pages need OCR?

The classify_pdf_mem function analyzes PDF structure in src/lib.rs and returns pages_needing_ocr as a list of zero‑indexed page numbers. It detects GID‑encoded fonts, broken encodings, and vector‑only content that native extraction cannot reliably decode. The classification runs in milliseconds without rendering pages.

What makes the layout detection "cheap" in pdf‑inspector?

Layout detectors operate on positioned text items, rectangles, and line geometry already extracted into memory by extract_positioned_text_from_doc. Because this geometric data exists in RAM, table and column detection in src/tables/ and src/extractor/mod.rs requires no additional PDF parsing or image rendering—unlike OCR‑based layout analysis that must rasterize pages first.

Can I use pdf‑inspector with custom OCR engines?

Yes. The extract_pages_markdown_mem function returns per‑page needs_ocr flags and page indices. Your application sends only flagged pages to any GPU OCR engine (PaddleOCR, Tesseract, cloud APIs), then merges the OCR text back into the markdown output. The crate intentionally separates extraction metadata from OCR execution for this flexibility.

Where does the actual OCR execution happen in pdf‑inspector?

The crate itself does not include OCR engines. According to the source code in src/lib.rs, it provides classification and extraction metadata that tells calling code which regions need OCR. The caller then routes those specific page images to their chosen GPU OCR implementation—enabling hybrid pipelines without bundling heavy OCR dependencies.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}