How pdf-inspector's Markdown Converter Detects Headings, Bold/Italic Text, Lists, Code Blocks, and URLs

pdf-inspector detects headings, bold/italic text, lists, code blocks, and URLs through a multi-stage pipeline that analyzes font statistics, visual cues, and textual patterns across five specialized Rust modules.

The firecrawl/pdf-inspector repository converts PDF content into clean, token-efficient Markdown by inferring semantic structure from visual presentation. Unlike tools that rely on embedded metadata, this converter rebuilds document hierarchy from raw text extraction using font analysis, spatial positioning, and pattern matching. This article explains exactly how each structural element gets recognized and rendered.

---

Overview of the Markdown Conversion Pipeline

The conversion process flows through three distinct phases:

1. Extraction – Raw text items are collected with their associated font properties
2. Analysis & Classification – Font statistics and visual patterns determine semantic roles
3. Rendering – Classified elements are emitted as proper Markdown syntax

The core logic lives in the markdown submodule under src/markdown/, with additional support from src/extractor/ for low-level PDF parsing.

---

Detecting Headings via Font Size Analysis

Headings are identified primarily through relative font size comparison rather than PDF structure trees alone.

Font Statistics Collection (analysis.rs)

The FontStats struct in src/markdown/analysis.rs computes page-level metrics:

• max_font_size – Largest font size encountered
• median_font_size – Median value for establishing baseline
• bold_ratio – Frequency of bold text usage

// From analysis.rs
pub struct FontStats {
    pub max_font_size: f32,
    pub median_font_size: f32,
    pub bold_ratio: f32,
}

The is_large_font function compares a line's font size against median_font_size * HEADING_FACTOR to identify potential headings.

Heading Level Assignment (heading.rs)

src/markdown/heading.rs contains the detect_heading function that maps relative font sizes to H1-H6 levels:

// Inside heading.rs
fn detect_heading(line: &TextLine, stats: &FontStats) -> Option<Heading> {
    if stats.is_large_font(line) {
        let level = stats.font_to_heading_level(line.font_size);
        Some(Heading { level, text: line.text.clone() })
    } else {
        None
    }
}

Larger fonts receive lower heading numbers (H1 for the largest). The system also falls back to PDF structure-tree roles (H1-H6) when explicitly available in the document metadata.

---

Detecting Bold and Italic Text

Text styling detection operates at the character level using flags embedded during font extraction.

Style Flag Propagation (extractor/fonts.rs → classify.rs)

Each TextItem carries boolean flags set during initial parsing:

• is_bold – Detected via font weight or family name patterns
• is_italic – Detected via font style properties

The apply_style function in src/markdown/classify.rs wraps text with appropriate Markdown delimiters:

// Inside classify.rs
fn apply_style(item: &TextItem) -> String {
    let mut txt = item.text.clone();
    if item.is_bold && item.is_italic {
        txt = format!("***{}***", txt);
    } else if item.is_bold {
        txt = format!("**{}**", txt);
    } else if item.is_italic {
        txt = format!("*{}*", txt);
    }
    txt
}

Combined bold-italic receives triple asterisks, matching standard Markdown conventions.

---

Detecting List Items

List detection combines character pattern matching with indentation depth measurement to handle nested structures.

Bullet and Ordered List Patterns

The is_list_item function in src/markdown/classify.rs recognizes:

• Bullet characters: ·, •, -, *
• Ordered prefixes: digit sequences followed by periods or parentheses (1., a), i.)

// Inside classify.rs
fn is_list_item(line: &TextLine) -> bool {
    let bullet = line.text.starts_with(['·','•','-','*']);
    let ordered = line.text.trim_start().starts_with(|c: char| c.is_ascii_digit()) 
                  && line.text.contains('.');
    bullet || ordered
}

Nesting via Indentation

Indentation depth is measured and used to group items into nested list structures. Consecutive items at the same depth form a single list block; increased depth triggers sub-list creation.

---

Detecting Code Blocks

Code block identification uses dual signals: font family classification and visual layout patterns.

Monospace Font Detection

The font cache in extractor/fonts.rs flags monospaced fonts. When consecutive lines share this property, they become candidates for code block treatment.

Indentation-Based Confirmation

Additional confirmation comes from measuring line indentation. Code blocks typically exhibit:

1. Consistent leading whitespace across multiple lines
2. Monospace font family
3. Absence of paragraph-style flow (short lines, uniform structure)

When verified, the block is emitted with triple backticks and appropriate language guessing if available.

---

Detecting and Rendering URLs

Hyperlink handling operates through two parallel mechanisms in src/extractor/links.rs.

PDF Link Annotations

Native PDF link annotations (URI actions) are extracted directly as PdfLink objects containing display text and destination URL. These render as standard Markdown links: [text](url).

Plain Text URL Detection

For URLs appearing as regular text, a lightweight regex pattern identifies candidates:

// Inside links.rs
fn extract_urls(text: &str) -> Vec<String> {
    let re = regex::Regex::new(r"https?://\S+").unwrap();
    re.find_iter(text).map(|m| m.as_str().to_string()).collect()
}

The pattern https?://\S+ captures HTTP and HTTPS URLs until whitespace boundaries.

---

Orchestrating Final Output in convert.rs

The src/markdown/convert.rs module coordinates the entire pipeline. It iterates over classified lines and delegates to specialized renderers:

• heading::render_heading – Emits # to ###### prefixed lines
• classify::render_styled_text – Applies inline formatting
• classify::render_list – Handles list syntax and nesting
• classify::render_code_block – Wraps with triple backticks

Spacing and line breaks are managed to ensure valid, readable Markdown output.

Complete Usage Example

// Example: Running the converter on a PDF file
use pdf_inspector::process_pdf_with_options;
let opts = pdf_inspector::ProcessOptions::default().with_json_output(true);
let result = process_pdf_with_options("sample.pdf", opts).unwrap();
// `result.markdown` now contains headings, styled text, lists, code blocks, and links.

---

Summary

• Headings – Detected via FontStats comparison in analysis.rs, mapped to levels in heading.rs
• Bold/Italic – Style flags from extractor/fonts.rs convert to ** and * wrappers in classify.rs
• Lists – Pattern matching for bullets/numbers plus indentation depth measurement in classify.rs
• Code blocks – Monospace font detection combined with consecutive indented line analysis
• URLs – Dual extraction from PDF annotations and regex matching in links.rs

The modular architecture separates concerns cleanly: statistics gathering, semantic classification, and final rendering each occupy distinct files with minimal coupling.

---

Frequently Asked Questions

Does pdf-inspector require PDFs to have embedded heading metadata?

No. While heading.rs can use structure-tree roles when present, the primary detection mechanism relies on relative font size analysis. Documents without semantic markup still convert accurately through visual inference.

How does the converter handle mixed bold and italic in the same text run?

The apply_style function checks both flags simultaneously. When is_bold && is_italic are true, it wraps with triple asterisks (***text***), following standard Markdown conventions for combined emphasis.

Can the code block detector distinguish between code and preformatted plain text?

Detection prioritizes monospace font family as the primary signal. The converter treats all monospace-indented sequences as code blocks. There's no semantic distinction between "code" and "preformatted text" in the current implementation—both render as fenced code blocks.

What happens if a URL appears inside bold or italic text?

URL extraction in links.rs operates on the raw text stream before styling application. During rendering in convert.rs, the Markdown structure places link syntax inside emphasis delimiters when both apply, producing valid nested markup like **[link](url)**.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}