How the PDF-Inspector Vision Module Fusion Pipeline Combines Native Text with OCR Results

The PDF-Inspector vision module's fusion pipeline merges native PDF text extraction with OCR results through a geometry-aware, quality-driven system that adaptively selects the best content source per page.

The fusion pipeline is the critical component in Firecrawl's pdf-inspector vision subsystem. Located in src/vision/fusion.rs, it intelligently combines native PDF extraction (PageMarkdown) with OCR output (OcrRun) to produce the most accurate Markdown representation of each page. This article explains exactly how the pipeline makes fusion decisions, preserves formatting, and maintains provenance tracking.

Core Fusion Pipeline Architecture

The fusion process operates as an 11-step pipeline that validates inputs, aligns page data, assesses content quality, and produces fused output with full traceability.

Step 1: Input Validation and Route Preparation

Before any fusion occurs, the pipeline validates configuration and prepares OCR routes. The validate_options function checks DPI settings and confidence thresholds (lines 95-105), while full_page_routes creates the default routing strategy (lines 40-46).

Routes determine how OCR content is applied:

• FullPage — Replace entire page with OCR output
• SupplementalRegions — Inject OCR only into specific geometric regions (e.g., tables or images)

Step 2: Page Alignment via BTreeMap Construction

The pipeline builds two ordered maps to synchronize native and OCR content:

// From src/vision/fusion.rs, lines 77-97
let native_numbers: BTreeMap<u32, &PageMarkdown>  // 1-indexed native pages
let ocr_by_page: BTreeMap<u32, &OcrPage>          // OCR by rendered page number

Errors are raised immediately for duplicate page numbers or mismatched counts. The equal_time_shares function (lines 83-93) also computes per-page render time allocations for provenance tracking.

Step 3: Per-Page Content Processing Loop

The main iteration (lines 108-140) handles each native page through several sub-operations:

1. Extract OCR text items via ocr_text_items (lines 84-135)
2. Discard unusable spans based on confidence and geometry filters
3. Determine routing strategy for the page
4. Generate OCR-only Markdown through ocr_page_to_markdown

Adaptive Content Selection: The choose_adaptive_content Function

The heart of the fusion logic resides in choose_adaptive_content (lines 37-84). This function implements quality-based arbitration between native and OCR sources.

Quality Assessment Criteria

Assessment	Function	Purpose
Native candidate quality	assess_native_candidate	Requires ≥8 alphanumeric chars, minimum quality score (lines 63-73)
OCR candidate quality	assess_text_candidate	Evaluates length, density, and line distribution (lines 47-81)
Content overlap	content_overlap	Detects duplication between sources
Mean OCR confidence	Built-in threshold	Compared against hosted_recommendation_confidence

Decision Outcomes

The function returns a PageContentSource enum variant:

• Native — Keep original PDF extraction
• Ocr — Replace with full-page OCR
• Fused — Merge both sources

When OCR confidence falls below hosted_recommendation_confidence (default 0.6) or adds no material novelty, the pipeline retains native content and flags a hosted pipeline recommendation for fallback processing (lines 44-55, 66-78).

Route-Specific Processing

Full-Page OCR Route

For pages routed to FullPage replacement:

// From src/vision/fusion.rs, lines 41-50
let markdown = to_markdown_from_items_with_rects_and_page_count(
    &items,
    page_width,
    page_height,
    page_count,
);
preserve_ocr_line_breaks(&markdown, &items)  // Restores column-aware breaks

The preserve_ocr_line_breaks function (lines 71-124) walks OCR span geometry, maps positions back to raw Markdown, and reinserts explicit double-newlines to maintain column structure that OCR typically loses.

Supplemental-Region Route

For SupplementalRegions routing:

1. Filter items to those inside specified PdfRect boundaries via items_inside_regions
2. Generate table Markdown using the same converter
3. Fallback handling — if no table detected, emit warning and retain native content (lines 26-40)

Native-OCR Fusion: The merge_native_and_ocr Algorithm

When both sources contain valuable, non-overlapping content, the pipeline executes token-level merging:

// From src/vision/fusion.rs, lines 5-53
pub fn merge_native_and_ocr(
    native: &str,
    ocr_items: &[TextItem],
    options: &OcrFusionOptions,
) -> Result<String, OcrFusionError>

The algorithm:

1. Tokenizes both native and OCR text
2. Removes duplicate content blocks through comparison
3. Appends only truly novel OCR fragments
4. Flags result as Fused in provenance

This prevents OCR from corrupting well-extracted native text while capturing content that PDF extraction missed (handwritten annotations, image-embedded text, etc.).

Provenance and Error Handling

FusedPageMarkdown Structure

Every output page carries complete metadata:

pub struct FusedPageMarkdown {
    pub page_number: u32,
    pub markdown: String,
    pub provenance: PageProvenance,  // Source, model, DPI, timing, warnings
}

pub struct PageProvenance {
    pub source: PageContentSource,           // Native | Ocr | Fused
    pub ocr_model: Option<String>,
    pub render_dpi: f32,
    pub ocr_time_share_ms: u64,
    pub warnings: Vec<String>,
    pub hosted_recommended: bool,           // Fallback recommendation flag
}

Error Types

All failure modes are enumerated in OcrFusionError (lines 13-55):

• Page number mismatches
• Duplicate page detection
• Invalid DPI or confidence configuration
• Routing errors

The public API surfaces these as Result<FusedPages, OcrFusionError> from fuse_ocr_pages and fuse_ocr_pages_adaptive.

Practical Usage Examples

Standard Fusion

use pdf_inspector::vision::{
    fuse_ocr_pages,
    OcrFusionOptions,
    PageMarkdown,
    OcrRun,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Native extraction from pdf-inspector core
    let native_pages = vec![
        PageMarkdown {
            page: 0,
            markdown: "Introduction\n\nParagraph text".to_string(),
            needs_ocr: false,
            ocr_reason: None,
        },
    ];

    // OCR from PDF-ium renderer
    let ocr_run = OcrRun {
        pages: vec![/* populated by vision renderer */],
        render_time_ms: 12,
        ocr_time_ms: 45,
    };

    let options = OcrFusionOptions::new()
        .render_dpi(300.0)
        .hosted_recommendation_confidence(0.6);

    let fused = fuse_ocr_pages(&native_pages, &ocr_run, 1, &options)?;

    for page in fused.pages {
        println!("Page {}: {:?}", page.page_number, page.provenance.source);
    }

    Ok(())
}

Adaptive Fusion with Native Candidates

use pdf_inspector::vision::fuse_ocr_pages_adaptive;
use std::collections::BTreeMap;

// Pre-assess trustworthy native content (e.g., from PDFium fallback)
let mut native_candidates = BTreeMap::new();
native_candidates.insert(
    1,
    assess_native_candidate(
        "Invoice total: $420.00\n".to_string(),
        NativeCandidateOrigin::Pdfium,
    ).unwrap(),
);

let fused = fuse_ocr_pages_adaptive(
    &native_pages,
    &ocr_run,
    1,
    &OcrFusionOptions::default(),
    &native_candidates,
)?;

Key Source Files

File	Purpose
src/vision/fusion.rs	Core fusion logic, quality assessment, adaptive decisions, provenance
src/vision/pipeline.rs	Orchestrates OCR execution and routes to fusion layer
src/vision/routing.rs	Routing enums (FullPage, SupplementalRegions)
src/vision/contracts.rs	Shared types (PageContentSource, PageProvenance)

Summary

• The fusion pipeline in src/vision/fusion.rs implements geometry-aware, quality-driven merging of native PDF text and OCR output
• choose_adaptive_content arbitrates between Native, Ocr, and Fused sources using multi-factor quality scoring
• Route-specific processing handles full-page replacement or targeted supplemental region injection
• preserve_ocr_line_breaks restores column-aware formatting that raw OCR loses
• Complete provenance tracking via FusedPageMarkdown and PageProvenance enables auditability and hosted fallback recommendations
• All operations return Result<T, OcrFusionError> for robust error handling

Frequently Asked Questions

What triggers the fusion pipeline to recommend the hosted pipeline?

The hosted_recommended flag is set when mean OCR confidence falls below hosted_recommendation_confidence (default 0.6) or when OCR content overlaps substantially with native text without adding novel information. This occurs in choose_adaptive_content lines 44-55 and 66-78, allowing callers to fall back to cloud-based processing when local OCR proves insufficient.

How does the pipeline prevent OCR from duplicating already-extracted text?

The merge_native_and_ocr function performs token-level comparison between native content and OCR items. It removes duplicate blocks and appends only genuinely novel OCR fragments. The result is flagged as Fused in provenance to indicate mixed-source origin.

Can the fusion pipeline handle partial-page OCR for specific regions like tables?

Yes. The SupplementalRegions route (defined in src/vision/routing.rs) restricts OCR to specific geometric PdfRect boundaries via items_inside_regions. This targets tables or image regions where native extraction fails while preserving reliable text elsewhere. If no table is detected, the pipeline emits a warning and retains native content.

What line-break issues does OCR cause and how are they fixed?

OCR engines lose original column and paragraph structure, producing run-together text. The preserve_ocr_line_breaks function (lines 71-124) analyzes OCR span geometry, maps positions back to raw Markdown coordinates, and reinserts explicit double-newlines to restore readable column-aware formatting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}