How pdf-inspector Performs Markdown Conversion: Heading Detection, List Parsing, and Code Blocks

pdf-inspector converts PDFs to clean Markdown through a multi-stage pipeline that classifies extracted text lines by font properties and structural cues, then emits proper heading, list, and code-block syntax.

The firecrawl/pdf-inspector library transforms raw PDF content into structured Markdown using a Rust-based pipeline. This article breaks down exactly how the tool detects headings, parses lists, and identifies code blocks—walking through the source files and algorithms responsible for each conversion stage.

---

Overview of the Markdown Conversion Pipeline

The conversion process lives in the src/markdown package and operates on a stream of PdfLine objects produced by the extractor. Each line carries text content, font metadata, and geometric positioning information from src/extractor/mod.rs.

The pipeline follows this sequence:

1. Pre-processing — normalizes fragmented text
2. Classification — determines line roles
3. Heading detection — builds hierarchical levels
4. List parsing — groups items by indentation and markers
5. Code-block detection — identifies monospaced runs
6. Conversion — emits final Markdown tokens
7. Post-processing — cleans artifacts and formats output

---

Step 1: Pre-processing Raw PDF Lines

Before classification begins, src/markdown/preprocess.rs handles common PDF extraction artifacts:

• Drop-cap merging — combines large decorative initials with their following text
• Fragmented heading joining — reunites heading lines broken across the page
• Whitespace normalization — collapses irregular spacing from justified text

This stage ensures downstream classifiers work with coherent text units rather than broken fragments.

---

Step 2: Line Classification by Content Type

The heart of the pipeline resides in src/markdown/classify.rs. Each PdfLine is examined for:

Criterion	Evaluated For
Font size	Heading hierarchy, emphasis
Font style (bold/italic/regular)	Semantic weight
Indentation level	List nesting, code blocks
Leading characters	Bullet markers, numeric prefixes
Punctuation patterns	Prose vs. code detection

The classifier assigns each line a structural role: heading candidate, list item, code line, or paragraph text.

---

Heading Detection and Hierarchy Building

In src/markdown/heading.rs, pdf-inspector constructs a document outline by analyzing font-size relationships:

• Larger fonts become higher-level headings (H1, H2)
• Sudden size drops indicate level demotion
• Consistent size patterns across pages establish baseline hierarchy

The algorithm builds a tree of heading levels (H1–H6) that maps directly to Markdown # syntax during conversion. Contextual cues—such as lines standing alone or preceding short paragraphs—help resolve ambiguities when multiple sizes appear.

---

List Parsing: Ordered and Unordered

List detection in src/markdown/classify.rs identifies items through:

Unordered lists:

• Bullet characters: •, -, *, ‣, ○
• Consistent leading indentation across consecutive items
• Visual grouping in the original layout

Ordered lists:

• Numeric prefixes: 1., (a), i., etc.
• Sequential validation when possible
• Indentation-based nesting depth tracking

The parser groups consecutive items into list structures and computes nesting levels by comparing indentation against parent contexts. This produces properly nested Markdown with correct - or 1. prefixes and appropriate indentation spacing.

---

Code Block Detection

Code blocks are recognized in src/markdown/classify.rs through heuristic analysis:

• Monospaced font families — the strongest signal
• Uniform indentation — ≥4 spaces or tab characters at line start
• Absence of sentence punctuation — few periods, question marks, or exclamation points
• Short line lengths — typical of formatted code vs. flowing prose
• Consecutive line runs — single code lines are preserved inline, runs of 2+ become fenced blocks

When a qualifying run is detected, the lines are wrapped in triple backticks (`) during the conversion phase. No language identifier is inferred—pdf-inspector outputs plain code fences without ```python style tags.

---

The Core Conversion Loop

src/markdown/convert.rs implements the final transformation:

// Simplified conceptual flow based on source structure
for line in classified_lines {
    match line.role {
        Heading(level) => output.push_str(&"#".repeat(level)),
        ListItem(marker, depth) => output.push_str(&"  ".repeat(depth) + marker),
        CodeBlock(lines) => {
            output.push_str("```\n");
            output.push_str(&lines.join("\n"));
            output.push_str("\n```");
        }
        Paragraph => output.push_str(&line.text),
    }
}

The converter respects reading order determined earlier by the document's layout detector—handling both newspaper-style columns and tabular arrangements without scrambling text sequence.

---

Post-processing and Output Cleanup

src/markdown/postprocess.rs removes artifacts that survive extraction:

Artifact	Cleanup Action
Dot leaders	Remove ......... from tables of contents
Hyphenation	Rejoin configura-\ntion → configuration
Stray page numbers	Strip isolated digits at column edges
URLs	Rewrite to canonical form

The final output can be emitted as:

# Plain Markdown to stdout

pdf2md document.pdf

# Saved to file

pdf2md document.pdf > output.md

# Structured JSON for downstream processing

pdf2md --json document.pdf > output.json

---

Library Usage Example

For programmatic integration:

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::ProcessOptions;

let opts = ProcessOptions::default();
let result = process_pdf_with_options("report.pdf", opts).unwrap();

// Access the Markdown string
println!("{}", result.markdown);

// Access structured data if JSON mode was enabled
println!("{}", result.json);

The ProcessOptions struct (exposed in src/lib.rs) allows configuration of table detection, OCR behavior, and other extraction parameters.

---

Summary

• pdf-inspector's Markdown conversion resides in src/markdown/ with specialized modules for preprocessing, classification, headings, conversion, and postprocessing
• Heading detection in src/markdown/heading.rs uses font-size hierarchy to build H1–H6 levels
• List parsing in src/markdown/classify.rs recognizes bullets, numbers, and indentation depth
• Code blocks are identified by monospaced fonts, uniform indentation, and low punctuation density
• The conversion loop in src/markdown/convert.rs emits standard Markdown syntax while preserving document reading order
• Post-processing cleans extraction artifacts before final output

---

Frequently Asked Questions

What determines whether a line becomes a heading vs. bold paragraph text?

pdf-inspector primarily uses relative font size and isolation from surrounding text. Headings appear significantly larger than body text and often stand alone on their lines. The src/markdown/heading.rs module builds a hierarchy across the document rather than judging lines in isolation—ensuring consistent level assignment even when absolute sizes vary between PDFs.

How does pdf-inspector handle nested lists with mixed bullet styles?

The classifier tracks indentation depth as the primary nesting signal. Bullet style changes (switching from - to * or •) at the same depth are treated as list continuations; indentation increases create nested sublists. The conversion preserves visual structure using standard Markdown indentation (two or four spaces per level).

Why are some code blocks not detected as fenced blocks?

Code detection requires multiple consecutive lines matching the heuristic (monospaced font, uniform indentation, minimal punctuation). Single lines of monospaced text remain inline. Additionally, syntax-highlighted code with color variations may fragment into multiple font runs that defeat the consecutive-line detection—src/markdown/preprocess.rs attempts to merge these but cannot always succeed.

Can the Markdown output include syntax highlighting for code blocks?

No—pdf-inspector outputs plain triple backticks without language identifiers. The PDF format rarely encodes programming language information reliably. Downstream consumers must apply syntax highlighting based on content analysis or manual annotation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}