Detecting and Removing Page Numbers from Extracted Text in pdf‑inspector: A Complete Guide

To remove page numbers from PDF text extraction in pdf‑inspector, enable MarkdownOptions::remove_page_numbers (defaults to true) and the pipeline will strip isolated numeric lines and explicit "Page N" patterns during markdown post‑processing.

The pdf‑inspector repository from Firecrawl provides a robust, three‑layer system for eliminating pagination noise without damaging legitimate numeric content. This guide examines the actual Rust implementation, explains when and how page numbers are detected, and shows you how to control the behavior programmatically.

How Page Number Detection Works in pdf‑inspector

Page number removal happens after the main layout analysis and markdown conversion stages. This sequencing is deliberate: it ensures that numbers embedded in sentences ("Page 42 explains the result") survive while standalone pagination markers disappear.

The Core Predicate: is_page_number_line

The foundation of detection lives in src/text_utils.rs. The is_page_number_line function implements a strict, two‑phase test:

1. Explicit pattern match – checks against is_explicit_page_number_expression
2. Keyword + digits validation – detects a leading "page" keyword followed only by digits and optional trailing whitespace

// src/text_utils.rs (lines 60-85)
pub(crate) fn is_page_number_line(text: &str) -> bool {
    // Implementation checks for explicit expressions and "page N" patterns
    // Returns true only for isolated page number lines
}

This conservatism protects content like "Page 42 explains" from false positives. The function returns true only when the entire line consists of nothing but a page number.

Markdown Post‑Processing: remove_page_numbers

The actual elimination occurs in src/markdown/postprocess.rs. The remove_page_numbers function scans the generated markdown line by line using the predicate above, but applies additional contextual safeguards:

• Isolated line check – the line must be surrounded by empty lines or markdown page‑break markers (---)
• Pre‑page‑break position – lines immediately preceding a page break are also candidates for removal

// src/markdown/postprocess.rs (lines 49-85)
fn remove_page_numbers(text: &str) -> String {
    // Examines each line with is_page_number_line
    // Drops candidates based on isolation or page-break adjacency
    // Returns cleaned markdown string
}

These guards prevent accidental deletion of numbers in tables of contents or enumerated lists. After removal, the cleaned string continues through the remaining post‑processing pipeline (dot‑leader collapsing, hyphenation repair, etc.).

Controlling Page Number Removal via MarkdownOptions

Stripping is optional and configured through MarkdownOptions::remove_page_numbers. The flag defaults to true as defined in src/markdown/mod.rs:

// src/markdown/mod.rs (lines 34-36)
pub struct MarkdownOptions {
    // ...
    pub remove_page_numbers: bool,
    // ...
}

This default behavior suits most extraction scenarios where pagination metadata adds noise. However, you can preserve page numbers when needed—for debugging, legal document review, or maintaining original pagination references.

Practical Code Examples

Default Behavior: Automatic Page Number Removal

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::markdown::MarkdownOptions;

let pdf_path = "example.pdf";
let options = MarkdownOptions::default(); // remove_page_numbers = true
let markdown = process_pdf_with_options(pdf_path, options).unwrap();
println!("{}", markdown);

The extracted markdown string will have standalone page numbers like "5" or "Page 7" removed, provided they appear in isolated positions.

Preserving Page Numbers

use pdf_inspector::markdown::MarkdownOptions;

let mut opts = MarkdownOptions::default();
opts.remove_page_numbers = false; // keep isolated page numbers
let markdown = process_pdf_with_options("example.pdf", opts).unwrap();

Set the flag to false when your downstream processing requires original pagination markers.

Direct Utility Access for Testing

use pdf_inspector::text_utils::is_page_number_line;

assert!(is_page_number_line("5"));                  // isolated digit
assert!(is_page_number_line("Page 42"));            // labeled format
assert!(!is_page_number_line("Page 42 explains"));  // embedded in sentence
assert!(!is_page_number_line("42 ways to win"));    // leading number with text

These assertions demonstrate the predicate's selectivity—essential for unit testing custom extraction logic.

Key Source Files and Their Roles

File	Responsibility	Critical Function/Struct
src/text_utils.rs	Low‑level pattern detection	is_page_number_line
src/markdown/postprocess.rs	Contextual removal execution	remove_page_numbers
src/markdown/mod.rs	Configuration interface	MarkdownOptions::remove_page_numbers
src/markdown/convert.rs	Pipeline orchestration	Calls clean_markdown which invokes post‑processing
src/markdown/preprocess.rs	Early header/footer handling	Precedes page‑number specific logic

Understanding this file structure helps when debugging extraction behavior or contributing improvements to the detection heuristics.

Summary

• Detection strictness – is_page_number_line in src/text_utils.rs uses pattern matching and keyword validation to avoid false positives on embedded numbers
• Contextual removal – remove_page_numbers in src/markdown/postprocess.rs only deletes isolated lines or pre‑page‑break candidates
• Configurable default – MarkdownOptions::remove_page_numbers defaults to true but can be disabled per‑extraction
• Pipeline sequencing – removal occurs after markdown conversion, protecting legitimate numeric content throughout layout analysis

Frequently Asked Questions

How can I verify that page numbers are being detected correctly?

Use the is_page_number_line utility directly from src/text_utils.rs. Pass candidate strings to confirm which patterns trigger detection before running full extraction.

Why are some page numbers still appearing in my output?

Numbers embedded in sentences ("see Page 12 for details") are intentionally preserved. Only isolated lines or lines immediately before page breaks qualify for removal. Check surrounding whitespace and markdown structure.

Does disabling remove_page_numbers affect other post‑processing steps?

No. The flag controls only the page‑number removal pass. Other post‑processing operations—dot‑leader collapsing, hyphenation fixes, header/footer normalization—continue unchanged.

Can I customize the page number pattern detection?

Currently pdf‑inspector uses hardcoded patterns in is_page_number_line and is_explicit_page_number_expression. For custom localization needs (e.g., "Seite 5" in German), you would need to modify src/text_utils.rs and rebuild.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}