How pdf-inspector Extracts H1-H6 Heading Roles from Tagged PDF Structure Trees

pdf-inspector reads the /StructTreeRoot dictionary in PDF files to obtain semantic H1-H6 tags, maps them to marked-content identifiers (MCIDs), and converts them directly into Markdown headings.

The firecrawl/pdf-inspector repository uses tagged PDF metadata to preserve document semantics during PDF-to-Markdown conversion. When a PDF contains a properly structured tree, heading roles are extracted authoritatively—bypassing the need for visual heuristics that guess heading levels from font sizes or positioning.

Parsing the PDF Structure Tree

The entry point is StructTree::from_doc in src/structure_tree.rs. This function traverses the PDF's /StructTreeRoot object and constructs an in-memory tree of StructElement nodes.

Key responsibilities of the parser:

• Resolves /RoleMap – Maps custom role names to standard PDF structure types (e.g., Heading1 → H1).
• Builds StructElement nodes – Each node carries a StructRole enum value.
• Handles standard heading roles – H, H1, H2, H3, H4, H5, H6 are all represented as distinct enum variants (StructRole::H1 through StructRole::H6).

// From src/structure_tree.rs, lines 18-39
// The StructRole enum includes all heading variants:
pub enum StructRole {
    Document,
    Part,
    Sect,
    // ... other roles ...
    H,      // Generic heading
    H1,
    H2,
    H3,
    H4,
    H5,
    H6,
    // ... additional roles ...
}

The parser recursively descends through parent-child relationships in the structure tree, preserving the hierarchical document organization encoded by the PDF creator.

Mapping MCIDs to Heading Roles

Once the structure tree is parsed, StructTree::mcid_to_roles generates a lookup table from marked-content identifiers (MCIDs) to their semantic roles. This enables downstream code to attach structural meaning to individual text spans.

The mapping is organized as:

Page number → { MCID → StructRole }

Key implementation details from src/structure_tree.rs (lines 43-52):

pub fn mcid_to_roles(
    &self,
    page_ids: &HashMap<u32, ObjectId>
) -> HashMap<u32, HashMap<i64, StructRole>> {
    // Returns per-page maps allowing fast MCID→role lookups
}

Why MCIDs matter: PDF content streams use MCIDs to tag individual drawing operations with structure element identifiers. By correlating MCIDs with StructRole values, pdf-inspector preserves semantic boundaries even when text runs are fragmented across multiple content stream operations.

Converting Structure Roles to Markdown Headings

The markdown pipeline processes heading roles in two stages.

Stage 1: Pre-processing Heading Levels

In src/markdown/preprocess.rs (lines 25-31), the detector translates StructRole values into numeric heading levels:

StructRole	Heading Level
H	1 (fallback)
H1	1
H2	2
H3	3
H4	4
H5	5
H6	6

The generic H role (without numeric suffix) defaults to level 1, ensuring backward compatibility with PDFs that use the older structure standard.

Stage 2: Markdown Generation

The converter in src/markdown/convert.rs (lines 507-513) emits proper Markdown syntax:

// From src/markdown/convert.rs
match heading_level {
    1 => format!("# {}\n", text),

    2 => format!("## {}\n", text),

    3 => format!("### {}\n", text),

    4 => format!("#### {}\n", text),

    5 => format!("##### {}\n", text),

    6 => format!("###### {}\n", text),

    _ => format!("{}\n", text), // No heading markup for unknown levels
}

This mapping guarantees that StructRole::H2 in the PDF becomes ## Heading Text in the output.

Practical Usage Examples

Basic PDF-to-Markdown Conversion

use pdf_inspector::{process_pdf_with_options, ProcessOptions};

let opts = ProcessOptions::default(); // Structure-tree parsing enabled by default
let md = process_pdf_with_options("annual-report.pdf", &opts)
    .expect("PDF parsing failed");

// Output contains #, ##, ### derived directly from PDF heading tags

println!("{}", md);

Accessing Raw MCID-to-Role Mappings

For custom post-processing or analysis:

use pdf_inspector::structure_tree::StructTree;
use lopdf::Document;

let doc = Document::load("annual-report.pdf").unwrap();
let tree = StructTree::from_doc(&doc).unwrap();
let page_ids = doc.get_pages().unwrap();

// Get the full MCID → role mapping
let mcid_roles = tree.mcid_to_roles(&page_ids);

// Find all H3 headings on page 2
if let Some(page_map) = mcid_roles.get(&2) {
    let h3_mcids: Vec<_> = page_map
        .iter()
        .filter_map(|(&mcid, role)| {
            if *role == pdf_inspector::structure_tree::StructRole::H3 {
                Some(mcid)
            } else {
                None
            }
        })
        .collect();
    
    println!("Page 2 H3 MCIDs: {:?}", h3_mcids);
}

Fallback Behavior When Structure Trees Are Absent

pdf-inspector prioritizes tagged PDF structure when available. If /StructTreeRoot is missing or incomplete:

1. Visual heuristics activate – Font size, weight, and positioning rules estimate heading boundaries.
2. Mixed handling – Partial structure trees are used where available; remaining content falls back to heuristics.

This hybrid approach maximizes semantic accuracy while maintaining robustness across the PDF ecosystem.

Key Source Files

File	Purpose
src/structure_tree.rs	Parses /StructTreeRoot, defines StructRole enum with H1-H6 variants, implements mcid_to_roles
src/markdown/preprocess.rs	Translates StructRole values to numeric heading levels
src/markdown/convert.rs	Emits Markdown heading syntax based on processed heading levels

Summary

• Structure tree parsing via StructTree::from_doc extracts authoritative heading roles from /StructTreeRoot
• MCID correlation maps structure elements to specific text runs in content streams
• Role-to-level translation converts StructRole::H1..H6 to numeric levels 1-6
• Markdown emission generates #..###### syntax based on definitive tagged PDF metadata
• Fallback mechanisms preserve functionality for untagged PDFs through visual analysis

Frequently Asked Questions

What happens when a PDF uses non-standard heading role names?

pdf-inspector resolves the /RoleMap dictionary in StructTree::from_doc to map custom names (e.g., Heading1, Title) to standard StructRole variants. If a mapping is absent, the role is treated as unknown and does not generate heading markup.

Can pdf-inspector extract heading hierarchy even when visual styling is misleading?

Yes. Because heading levels derive from the tagged PDF structure tree rather than font metrics, documents with unusual styling (small H1s, large body text) still convert correctly. The structure tree encodes author-intended semantics independent of rendering parameters.

How does the MCID mapping handle multi-page documents?

mcid_to_roles returns a HashMap<u32, HashMap<i64, StructRole>> where the outer key is the 1-based page number. This isolation prevents MCID collisions across pages, since MCIDs are only unique within a single page's content stream.

Is structure-tree parsing enabled by default?

Yes. ProcessOptions::default() enables structure tree extraction. Disabling it requires explicit configuration and forces reliance on visual heuristics for heading detection.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}