# Understanding the pdf-inspector Processing Pipeline Architecture: A Deep Dive into the 6-Stage Rust Implementation

> Explore the 6-stage Rust architecture of the pdf-inspector processing pipeline. Discover how it detects PDF types extracts text analyzes layouts and converts content to Markdown.

- Repository: [Firecrawl/pdf-inspector](https://github.com/firecrawl/pdf-inspector)
- Tags: architecture
- Published: 2026-08-13

---

**The pdf-inspector processing pipeline uses a six-stage Rust architecture that detects PDF types, extracts positioned text with spatial coordinates, analyzes complex layouts including tables and columns, intelligently flags pages requiring OCR, and converts structured content into clean Markdown.**

The `firecrawl/pdf-inspector` repository implements a modular, testable pipeline designed to handle everything from simple text-based PDFs to scanned documents and mixed-layout publications. Each stage operates as a discrete unit with pure functions, enabling developers to run the full extraction workflow or invoke individual phases like detection-only analysis.

## The Six Stages of the pdf-inspector Processing Pipeline

The architecture deliberately separates concerns into six distinct phases, each exposed through specific modules in the source tree.

### Stage 1: PDF-Type Detection

Before extracting content, the pipeline classifies the document to optimize downstream processing. In [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs), the system samples content streams to count text operators, images, vector-drawn text, and font-decoding capabilities. It categorizes the PDF as **TextBased**, **Scanned**, **ImageBased**, or **Mixed**, while building a list of pages requiring OCR.

This classification prevents wasted computation on image-only pages and ensures text-heavy documents skip unnecessary computer vision steps.

### Stage 2: Text and Position Extraction

Once classified, [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) loads the document once using `lopdf::Document` and traverses each page’s content streams—including Form XObjects—to extract every `Tj` and `TJ` operator. The extractor produces `TextItem` structs with XY coordinates, collects bounding rectangles (`PdfRect`), and records line segments (`PdfLine`).

This stage also identifies pages using GID-encoded fonts that may require special handling, storing this metadata for the OCR decision phase.

### Stage 3: Layout Analysis

The [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) module analyzes the positioned items to reconstruct the reading order. It detects multi-column layouts, newspaper-style formatting, and pre-masks page numbers to prevent them from interrupting content flow.

Table detection runs three sequential strategies: rectangle-based analysis, line-based detection, and heuristic fallback. The result is a `LayoutComplexity` description that guides the final Markdown rendering.

### Stage 4: OCR Decision Logic

Combining signals from the detection phase and layout analysis, the pipeline flags specific pages for optical character recognition. The [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) functions `page_ocr_reasons` and `used_fonts_have_identity_h_no_tounicode` evaluate template images, vector text presence, and undecodable fonts.

Pages receive specific `ocr_reasons_by_page` entries such as `scanned` or `suspected_garbled_text`, allowing downstream systems to apply targeted OCR only where necessary.

### Stage 5: Markdown Conversion

The [`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs) module transforms positioned items and layout metadata into linear, token-efficient Markdown. It applies language-aware heuristics for heading detection (using font-size statistics from [`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs)), merges list items, and handles typographic edge cases like drop-caps.

The converter respects the layout complexity determined earlier, ensuring tables render properly and columnar text flows logically.

### Stage 6: Public API Interface

[`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) exposes high-level convenience functions that orchestrate the entire pipeline or expose individual stages. The primary entry points include `process_pdf` for full extraction, `detect_pdf` for metadata-only classification, `extract_pages_markdown` for selective page processing, and `extract_text_in_regions_mem` for bounding-box-based extraction on tagged PDFs.

## Data Flow and Core Abstractions

The pipeline follows a pure functional flow where each stage returns structured results consumed by the next:

```

process_pdf → detect_pdf_type (detector) → load_document_once → 
extract_positioned_text (extractor) → 
layout detection (layout) + table detection (tables) → 
OCR page list (detector) → 
to_markdown (markdown) → PdfProcessResult

```

Each step yields strongly-typed results such as `PdfProcessResult`, `PdfClassification`, and `PagesExtractionResult`. This design keeps the pipeline testable and allows developers to inject custom logic between stages without modifying core extraction code.

## Practical API Usage Examples

The public API supports both high-level full-document processing and granular, region-specific extraction.

Run the complete pipeline with full detection and Markdown output:

```rust
// Full detection + extraction + markdown (default mode)
let result = pdf_inspector::process_pdf("report.pdf")?;
println!("PDF type: {:?}, pages: {}", result.pdf_type, result.page_count);
if let Some(md) = result.markdown {
    println!("Markdown output:\n{md}");
}

```

For quick metadata assessment without text extraction:

```rust
// Fast metadata‑only detection (no text extraction)
let meta = pdf_inspector::detect_pdf("scanned.pdf")?;
println!("Detected as: {:?}, OCR recommended: {}", meta.pdf_type, meta.ocr_recommended);

```

Extract specific pages for hybrid OCR workflows:

```rust
// Extract only page 3 as Markdown (useful for hybrid OCR pipelines)
let pages_md = pdf_inspector::extract_pages_markdown("large.pdf", Some(&[2]))?;
println!("Page 3 markdown:\n{}", pages_md.pages[0].markdown);

```

Perform region-based extraction using PDF point coordinates:

```rust
// Region‑based extraction – give bounding boxes in PDF points
let regions = vec![(0, vec![[50.0, 500.0, 300.0, 700.0]])]; // page 0, one box
let texts = pdf_inspector::extract_text_in_regions_mem(&buffer, &regions)?;
println!("Region text: {}", texts[0].regions[0].text);

```

## Key Source Files and Module Organization

The architecture spans multiple specialized modules:

- **[`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)**: Public façade and option builders
- **[`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs)**: Classification logic and OCR reason generation
- **[`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs)**: Extraction orchestration and form-field handling
- **[`src/extractor/content_stream.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/content_stream.rs)**: Low-level content stream parsing and font tracking
- **[`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs)**: Column detection and reading-order heuristics
- **[`src/markdown/convert.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs)**: Core Markdown rendering engine
- **[`src/markdown/analysis.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs)**: Font-size statistics for structure detection
- **`src/tables/*.rs`**: Three-stage table detection and formatting
- **[`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs)**: CMap handling for CID font decoding
- **[`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs)**: CJK, RTL, and glyph normalization utilities

## Summary

- The **pdf-inspector processing pipeline** implements a six-stage architecture: detection, extraction, layout analysis, OCR decision, Markdown conversion, and API exposure.
- Each stage resides in dedicated modules like [`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs) and [`extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/extractor/mod.rs), enabling unit testing and selective execution.
- The system classifies PDFs as TextBased, Scanned, ImageBased, or Mixed before processing, optimizing resource usage.
- Table detection employs three strategies (rectangle, line, heuristic) in [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs) to handle complex layouts.
- The pipeline returns structured results (`PdfProcessResult`) and supports both full-document and region-specific extraction via [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs).

## Frequently Asked Questions

### How does pdf-inspector decide whether a page needs OCR?

The pipeline analyzes multiple signals in [`src/detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) through functions like `page_ocr_reasons` and `used_fonts_have_identity_h_no_tounicode`. It checks for template images, vector text absence, and fonts lacking proper ToUnicode CMaps. When pages show these characteristics, they receive entries in `ocr_reasons_by_page` such as `scanned` or `suspected_garbled_text`, triggering OCR recommendations without unnecessary processing of clean text pages.

### What layout structures can the pipeline detect?

According to the source code in [`src/extractor/layout.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs), the system detects multi-column layouts, newspaper-style reading orders, and complex table structures. It runs three detection strategies sequentially—rectangle-based, line-based, and heuristic—to identify tabular data. The layout analyzer also pre-masks page numbers and handles edge cases like drop-caps to ensure logical reading order in the final Markdown output.

### Can I run only specific stages of the pipeline instead of the full extraction?

Yes. While `process_pdf` runs the complete workflow, the public API in [`src/lib.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs) exposes discrete functions like `detect_pdf` for classification-only analysis and `extract_pages_markdown` for selective page processing. The modular design of [`detector.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/detector.rs) and the extractor modules allows developers to import and execute individual stages, consuming intermediate structures like `PdfClassification` or `PagesExtractionResult` without running subsequent phases.

### How does the pipeline handle different PDF font encodings?

The extractor in [`src/extractor/mod.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/mod.rs) identifies GID-encoded fonts and problematic CMap encodings during text extraction. The [`src/tounicode.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/tounicode.rs) module provides fallback font decoding for CID fonts, while [`src/text_utils.rs`](https://github.com/firecrawl/pdf-inspector/blob/main/src/text_utils.rs) handles CJK character detection, RTL (right-to-left) text, and glyph normalization. These utilities ensure accurate text extraction even when PDFs use non-standard or legacy font encodings that would otherwise produce garbled output.