# How the PDF-Inspector Vision Module Fusion Pipeline Combines Native Text with OCR Results

> Discover how the PDF-Inspector vision module fuses native text with OCR results using a geometry-aware, quality-driven system. Select the best content source per page for superior accuracy.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: internals
- Published: 2026-09-02

---

**The PDF-Inspector vision module's fusion pipeline merges native PDF text extraction with OCR results through a geometry-aware, quality-driven system that adaptively selects the best content source per page.**

The **fusion pipeline** is the critical component in [Firecrawl's pdf-inspector](https://github.com/firecrawl/pdf-inspector) vision subsystem. Located in [`src/vision/fusion.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/fusion.rs), it intelligently combines **native PDF extraction** (`PageMarkdown`) with **OCR output** (`OcrRun`) to produce the most accurate Markdown representation of each page. This article explains exactly how the pipeline makes fusion decisions, preserves formatting, and maintains provenance tracking.

## Core Fusion Pipeline Architecture

The fusion process operates as an 11-step pipeline that validates inputs, aligns page data, assesses content quality, and produces fused output with full traceability.

### Step 1: Input Validation and Route Preparation

Before any fusion occurs, the pipeline validates configuration and prepares OCR routes. The `validate_options` function checks DPI settings and confidence thresholds (lines 95-105), while `full_page_routes` creates the default routing strategy (lines 40-46).

Routes determine how OCR content is applied:

- **`FullPage`** — Replace entire page with OCR output
- **`SupplementalRegions`** — Inject OCR only into specific geometric regions (e.g., tables or images)

### Step 2: Page Alignment via BTreeMap Construction

The pipeline builds two ordered maps to synchronize native and OCR content:

```rust
// From src/vision/fusion.rs, lines 77-97
let native_numbers: BTreeMap<u32, &PageMarkdown>  // 1-indexed native pages
let ocr_by_page: BTreeMap<u32, &OcrPage>          // OCR by rendered page number

```

Errors are raised immediately for duplicate page numbers or mismatched counts. The `equal_time_shares` function (lines 83-93) also computes per-page render time allocations for provenance tracking.

### Step 3: Per-Page Content Processing Loop

The main iteration (lines 108-140) handles each native page through several sub-operations:

1. **Extract OCR text items** via `ocr_text_items` (lines 84-135)
2. **Discard unusable spans** based on confidence and geometry filters
3. **Determine routing strategy** for the page
4. **Generate OCR-only Markdown** through `ocr_page_to_markdown`

## Adaptive Content Selection: The `choose_adaptive_content` Function

The heart of the fusion logic resides in `choose_adaptive_content` (lines 37-84). This function implements **quality-based arbitration** between native and OCR sources.

### Quality Assessment Criteria

| Assessment | Function | Purpose |
|------------|----------|---------|
| Native candidate quality | `assess_native_candidate` | Requires ≥8 alphanumeric chars, minimum quality score (lines 63-73) |
| OCR candidate quality | `assess_text_candidate` | Evaluates length, density, and line distribution (lines 47-81) |
| Content overlap | `content_overlap` | Detects duplication between sources |
| Mean OCR confidence | Built-in threshold | Compared against `hosted_recommendation_confidence` |

### Decision Outcomes

The function returns a **`PageContentSource`** enum variant:

- **`Native`** — Keep original PDF extraction
- **`Ocr`** — Replace with full-page OCR
- **`Fused`** — Merge both sources

When OCR confidence falls below `hosted_recommendation_confidence` (default 0.6) or adds no material novelty, the pipeline retains native content and flags a **hosted pipeline recommendation** for fallback processing (lines 44-55, 66-78).

## Route-Specific Processing

### Full-Page OCR Route

For pages routed to `FullPage` replacement:

```rust
// From src/vision/fusion.rs, lines 41-50
let markdown = to_markdown_from_items_with_rects_and_page_count(
    &items,
    page_width,
    page_height,
    page_count,
);
preserve_ocr_line_breaks(&markdown, &items)  // Restores column-aware breaks

```

The `preserve_ocr_line_breaks` function (lines 71-124) walks OCR span geometry, maps positions back to raw Markdown, and reinserts explicit double-newlines to maintain column structure that OCR typically loses.

### Supplemental-Region Route

For `SupplementalRegions` routing:

1. **Filter items** to those inside specified `PdfRect` boundaries via `items_inside_regions`
2. **Generate table Markdown** using the same converter
3. **Fallback handling** — if no table detected, emit warning and retain native content (lines 26-40)

## Native-OCR Fusion: The `merge_native_and_ocr` Algorithm

When both sources contain valuable, non-overlapping content, the pipeline executes **token-level merging**:

```rust
// From src/vision/fusion.rs, lines 5-53
pub fn merge_native_and_ocr(
    native: &str,
    ocr_items: &[TextItem],
    options: &OcrFusionOptions,
) -> Result<String, OcrFusionError>

```

The algorithm:

1. Tokenizes both native and OCR text
2. Removes duplicate content blocks through comparison
3. Appends only **truly novel OCR fragments**
4. Flags result as `Fused` in provenance

This prevents OCR from corrupting well-extracted native text while capturing content that PDF extraction missed (handwritten annotations, image-embedded text, etc.).

## Provenance and Error Handling

### FusedPageMarkdown Structure

Every output page carries complete metadata:

```rust
pub struct FusedPageMarkdown {
    pub page_number: u32,
    pub markdown: String,
    pub provenance: PageProvenance,  // Source, model, DPI, timing, warnings
}

pub struct PageProvenance {
    pub source: PageContentSource,           // Native | Ocr | Fused
    pub ocr_model: Option<String>,
    pub render_dpi: f32,
    pub ocr_time_share_ms: u64,
    pub warnings: Vec<String>,
    pub hosted_recommended: bool,           // Fallback recommendation flag
}

```

### Error Types

All failure modes are enumerated in `OcrFusionError` (lines 13-55):

- Page number mismatches
- Duplicate page detection
- Invalid DPI or confidence configuration
- Routing errors

The public API surfaces these as `Result<FusedPages, OcrFusionError>` from `fuse_ocr_pages` and `fuse_ocr_pages_adaptive`.

## Practical Usage Examples

### Standard Fusion

```rust
use pdf_inspector::vision::{
    fuse_ocr_pages,
    OcrFusionOptions,
    PageMarkdown,
    OcrRun,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Native extraction from pdf-inspector core
    let native_pages = vec![
        PageMarkdown {
            page: 0,
            markdown: "Introduction\n\nParagraph text".to_string(),
            needs_ocr: false,
            ocr_reason: None,
        },
    ];

    // OCR from PDF-ium renderer
    let ocr_run = OcrRun {
        pages: vec![/* populated by vision renderer */],
        render_time_ms: 12,
        ocr_time_ms: 45,
    };

    let options = OcrFusionOptions::new()
        .render_dpi(300.0)
        .hosted_recommendation_confidence(0.6);

    let fused = fuse_ocr_pages(&native_pages, &ocr_run, 1, &options)?;

    for page in fused.pages {
        println!("Page {}: {:?}", page.page_number, page.provenance.source);
    }

    Ok(())
}

```

### Adaptive Fusion with Native Candidates

```rust
use pdf_inspector::vision::fuse_ocr_pages_adaptive;
use std::collections::BTreeMap;

// Pre-assess trustworthy native content (e.g., from PDFium fallback)
let mut native_candidates = BTreeMap::new();
native_candidates.insert(
    1,
    assess_native_candidate(
        "Invoice total: $420.00\n".to_string(),
        NativeCandidateOrigin::Pdfium,
    ).unwrap(),
);

let fused = fuse_ocr_pages_adaptive(
    &native_pages,
    &ocr_run,
    1,
    &OcrFusionOptions::default(),
    &native_candidates,
)?;

```

## Key Source Files

| File | Purpose |
|------|---------|
| [`src/vision/fusion.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/fusion.rs) | Core fusion logic, quality assessment, adaptive decisions, provenance |
| [`src/vision/pipeline.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/pipeline.rs) | Orchestrates OCR execution and routes to fusion layer |
| [`src/vision/routing.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/routing.rs) | Routing enums (`FullPage`, `SupplementalRegions`) |
| [`src/vision/contracts.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/contracts.rs) | Shared types (`PageContentSource`, `PageProvenance`) |

## Summary

- The fusion pipeline in [`src/vision/fusion.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/fusion.rs) implements **geometry-aware, quality-driven merging** of native PDF text and OCR output
- **`choose_adaptive_content`** arbitrates between `Native`, `Ocr`, and `Fused` sources using multi-factor quality scoring
- **Route-specific processing** handles full-page replacement or targeted supplemental region injection
- **`preserve_ocr_line_breaks`** restores column-aware formatting that raw OCR loses
- **Complete provenance tracking** via `FusedPageMarkdown` and `PageProvenance` enables auditability and hosted fallback recommendations
- All operations return `Result<T, OcrFusionError>` for robust error handling

## Frequently Asked Questions

### What triggers the fusion pipeline to recommend the hosted pipeline?

The `hosted_recommended` flag is set when mean OCR confidence falls below `hosted_recommendation_confidence` (default 0.6) or when OCR content overlaps substantially with native text without adding novel information. This occurs in `choose_adaptive_content` lines 44-55 and 66-78, allowing callers to fall back to cloud-based processing when local OCR proves insufficient.

### How does the pipeline prevent OCR from duplicating already-extracted text?

The `merge_native_and_ocr` function performs **token-level comparison** between native content and OCR items. It removes duplicate blocks and appends only genuinely novel OCR fragments. The result is flagged as `Fused` in provenance to indicate mixed-source origin.

### Can the fusion pipeline handle partial-page OCR for specific regions like tables?

Yes. The `SupplementalRegions` route (defined in [`src/vision/routing.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/vision/routing.rs)) restricts OCR to specific geometric `PdfRect` boundaries via `items_inside_regions`. This targets tables or image regions where native extraction fails while preserving reliable text elsewhere. If no table is detected, the pipeline emits a warning and retains native content.

### What line-break issues does OCR cause and how are they fixed?

OCR engines lose original column and paragraph structure, producing run-together text. The `preserve_ocr_line_breaks` function (lines 71-124) analyzes OCR span geometry, maps positions back to raw Markdown coordinates, and reinserts explicit double-newlines to restore readable column-aware formatting.