pdf-inspector vs LiteParse, OpenDataLoader & PyMuPDF4LLM: Benchmark Comparison on TEDS, MHS, and NID Scores

pdf-inspector achieves the highest overall semantic quality score (0.875) and fastest processing time (0.470s for 200 PDFs) compared to LiteParse (0.873/0.750s), OpenDataLoader (0.831/2.569s), and PyMuPDF4LLM (0.735/17.117s) on the opendataloader-bench corpus.

The pdf-inspector repository is a Rust-native PDF classification and extraction library designed for local, OCR-free processing. Built by firecrawl/pdf-inspector, it prioritizes speed and semantic accuracy through single-load document sharing, fast content-stream sampling, and a dual-mode table detector. The results below demonstrate how these architectural decisions translate into measurable performance advantages across three critical metrics: TEDS (Table Extraction Distance Score), MHS (Markdown Heading Score), and NID (Normalized Insertion Distance).

---

Complete Benchmark Results

The evaluation was conducted on 200 PDFs from the opendataloader-bench corpus. Each engine was measured across four dimensions:

Engine	Overall	Reading Order (NID)	Tables (TEDS)	Headings (MHS)	Speed (200 docs)
pdf-inspector	0.875	0.915	0.814	0.788	0.470 s
LiteParse	0.873	0.913	0.693	0.811	0.750 s
OpenDataLoader	0.831	0.902	0.489	0.739	2.569 s
PyMuPDF4LLM	0.735	0.886	0.401	0.424	17.117 s

Source: Benchmark table in README.md

---

NID Score: Reading Order Reconstruction

pdf-inspector dominates reading-order accuracy with a 0.915 NID score, narrowly edging LiteParse (0.913) while significantly outperforming OpenDataLoader (0.902) and PyMuPDF4LLM (0.886).

This metric measures how well an engine reconstructs the logical flow of text across columns, sidebars, and complex layouts. pdf-inspector's advantage stems from its column-aware layout engine implemented in [src/extractor/layout.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs). The crate analyzes geometric relationships between text blocks to determine proper reading sequence, rather than relying on raw PDF content stream order—which often places floating elements like headers and footers in arbitrary positions.

The 0.002-point margin over LiteParse is meaningful at this performance tier: it indicates pdf-inspector handles edge cases in multi-column academic papers and financial reports more reliably.

---

TEDS Score: Table Extraction Quality

pdf-inspector achieves 0.814 TEDS—17% higher than LiteParse (0.693) and more than double OpenDataLoader (0.489) and PyMuPDF4LLM (0.401).

TEDS evaluates structural fidelity: whether extracted tables maintain correct row/column relationships, merged cells, and hierarchical headers. pdf-inspector's substantial lead comes from two complementary strategies in [src/tables/detect_rects.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/detect_rects.rs):

• Rectangle-based detection: Identifies table boundaries by analyzing graphic drawing operations (rectangles, lines) that visually define cell structures
• Heuristic fallback: When explicit graphics are absent, falls back to whitespace and alignment analysis

This dual-mode approach explains the 0.121-point gap over LiteParse, which relies primarily on heuristic methods. For data-heavy PDFs—financial statements, research tables, CSV-like reports—this translates to dramatically fewer garbled or misaligned outputs.

---

MHS Score: Heading Detection Accuracy

LiteParse narrowly leads MHS at 0.811 versus pdf-inspector's 0.788, though pdf-inspector still outperforms OpenDataLoader (0.739) and PyMuPDF4LLM (0.424) by substantial margins.

MHS measures how accurately engines identify document hierarchy (H1, H2, H3) and convert it to Markdown heading syntax. LiteParse's aggressive heading heuristics—likely over-generous elevation of bold or larger text—give it a slight edge in raw detection count.

pdf-inspector's approach in [src/markdown/analysis.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs) uses font-size tier clustering: grouping text elements by normalized size buckets, then applying contextual rules to distinguish true headings from emphasis or title styling. This produces more conservative but semantically precise output, reducing false-positive headings that disrupt document navigation.

The 0.023-point deficit is a deliberate quality-speed tradeoff—pdf-inspector prioritizes heading reliability over aggressive detection.

---

Processing Speed Analysis

pdf-inspector processes 200 PDFs in 0.470 seconds—1.6× faster than LiteParse, 5.5× faster than OpenDataLoader, and 36× faster than PyMuPDF4LLM.

Engine	Time per 200 PDFs	Relative Speed
pdf-inspector	0.470 s	Baseline
LiteParse	0.750 s	1.6× slower
OpenDataLoader	2.569 s	5.5× slower
PyMuPDF4LLM	17.117 s	36.4× slower

This throughput advantage enables real-time PDF pipelines: batch processing thousands of documents per hour without queue backlogs. The Rust implementation's zero-cost abstractions and memory-efficient content-stream sampling avoid the Python interpreter overhead that limits PyMuPDF4LLM and OpenDataLoader.

---

How to Run pdf-inspector

The library exposes identical semantics across Python, Node.js, and Rust via native bindings.

Python

import pdf_inspector

result = pdf_inspector.process_pdf("sample.pdf")
print("PDF type:", result.pdf_type)          # e.g. "text_based"

print("Reading order score:", result.reading_order_score)  # NID-like metric

print("Markdown output:\n", result.markdown)

Node.js (N-API)

import { readFileSync } from "fs";
import { processPdf } from "@firecrawl/pdf-inspector";

const pdf = readFileSync("sample.pdf");
const { pdfType, markdown } = processPdf(pdf);

console.log("PDF type:", pdfType);     // "TextBased", "Scanned", …
console.log(markdown);

Rust (direct crate)

use pdf_inspector::process_pdf;

let result = process_pdf("sample.pdf")?;
println!("PDF type: {:?}", result.pdf_type);
if let Some(md) = result.markdown {
    println!("{}", md);
}

---

Key Implementation Files

Understanding the source architecture explains pdf-inspector's benchmark performance:

File	Role
[src/lib.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/lib.rs)	Public API entry points: process_pdf, classify_pdf
[src/detector.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs)	Fast PDF-type classification (TextBased / Scanned / Hybrid)
[src/extractor/layout.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/extractor/layout.rs)	Column detection and reading-order reconstruction (drives NID)
[src/tables/detect_rects.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/detect_rects.rs)	Rectangle-based table detection with heuristic fallback (drives TEDS)
[src/markdown/analysis.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/analysis.rs)	Font-size tier clustering for heading identification (drives MHS)
[src/markdown/convert.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/markdown/convert.rs)	Final Markdown generation pipeline
[docs/benchmarking.md](https://github.com/firecrawl/pdf-inspector/blob/main/docs/benchmarking.md)	Reproducible benchmark harness

---

Summary

• pdf-inspector leads overall semantic quality (0.875) with top scores in reading order (NID: 0.915) and table extraction (TEDS: 0.814)
• TEDS advantage is decisive: 0.814 vs. 0.693 (LiteParse), critical for data-heavy document workflows
• Speed dominates all competitors: 0.470s processing time enables real-time pipelines
• MHS tradeoff is minor: 0.788 vs. 0.811 (LiteParse) reflects conservative, accurate heading detection versus aggressive heuristics
• Rust-native architecture delivers measurable efficiency gains over Python-based alternatives

For production systems requiring fast, high-fidelity Markdown conversion without OCR, pdf-inspector's benchmark results demonstrate clear superiority.

---

Frequently Asked Questions

What do TEDS, MHS, and NID measure in PDF extraction?

TEDS (Table Extraction Distance Score) measures structural accuracy of detected tables—whether rows, columns, and merged cells are correctly identified. MHS (Markdown Heading Score) evaluates hierarchical heading detection (H1, H2, H3) accuracy. NID (Normalized Insertion Distance) quantifies reading-order reconstruction: how well the engine sequences text blocks into logical document flow. These three metrics together capture semantic document understanding beyond raw text extraction.

Why does pdf-inspector outperform PyMuPDF4LLM so dramatically on TEDS?

pdf-inspector's 0.814 vs. 0.401 TEDS gap stems from architectural differences. PyMuPDF4LLM relies on PyMuPDF's basic table detection, which primarily uses text alignment heuristics. pdf-inspector implements dual-mode detection in [src/tables/detect_rects.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/tables/detect_rects.rs): it first searches for explicit graphic rectangles that define cell boundaries (common in professionally typeset PDFs), then falls back to whitespace analysis. This captures tables with complex visual formatting that alignment-only methods miss.

Is pdf-inspector suitable for scanned PDFs requiring OCR?

No—pdf-inspector explicitly targets OCR-free, text-based PDF processing. Its [src/detector.rs](https://github.com/firecrawl/pdf-inspector/blob/main/src/detector.rs) classifies PDFs as "TextBased", "Scanned", or "Hybrid" and optimizes extraction for native text content. For scanned documents, engines integrating Tesseract or cloud OCR APIs remain necessary. The benchmark comparison applies only to text-based PDFs where all four engines operate on comparable inputs.

How can I reproduce these benchmark results?

The repository includes a reproducible harness documented in [docs/benchmarking.md](https://github.com/firecrawl/pdf-inspector/blob/main/docs/benchmarking.md). The evaluation uses the opendataloader-bench corpus of 200 diverse PDFs across academic, financial, and technical domains. Each engine processes identical inputs, with TEDS/MHS/NID scores computed against ground-truth annotations. Run times are measured as wall-clock duration for complete batch processing on standardized hardware.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}