How to Extract Semantic Structure from Tagged PDFs Using pdf‑inspector

pdf‑inspector reads the /StructTreeRoot hierarchy of tagged PDFs to map every semantic element—headings, paragraphs, tables, and figures—to its corresponding text content via Marked Content IDs (MCIDs), enabling conversion to structured Markdown or JSON.

The firecrawl/pdf‑inspector Rust library provides a robust pipeline for parsing PDF tagging information according to ISO standards. Unlike simple text extraction tools that lose document hierarchy, pdf‑inspector preserves semantic roles by joining the PDF’s structure tree with raw content streams, allowing you to convert tagged PDFs into Markdown that respects original heading levels, list structures, and table semantics.

Understanding Tagged PDFs and the Structure Tree

Tagged PDFs embed a structure tree (located at /StructTreeRoot in the PDF catalog) that defines logical document elements independently from visual layout. Each node in this tree represents a semantic component—such as headings (H1–H6), paragraphs (P), tables (Table), or table cells (TD)—and references specific content through Marked Content IDs (MCIDs).

When pdf‑inspector processes a document, it resolves these references against the page’s content streams to produce a mapping between text content and its structural role. This approach allows the library to handle custom role mappings defined in the PDF’s /RoleMap dictionary while gracefully falling back to untagged extraction when no structure tree exists.

The Three-Step Extraction Pipeline

The extraction process implemented in firecrawl/pdf‑inspector follows a distinct three-phase architecture to separate parsing from content retrieval.

Parse the Structure Tree

The entry point StructTree::from_doc in src/structure_tree.rs (line 301) traverses the /StructTreeRoot dictionary to build an in-memory tree of StructElement nodes. Each node stores:

• A StructRole (standard or custom-mapped)
• Optional alternate text and language attributes
• A list of MarkedContentRef leaf references pointing to specific content instances

If the PDF lacks tagging information, this function returns None, allowing callers to proceed with unstructured text extraction.

Build the MCID-to-Role Lookup

Once the tree is parsed, StructTree::mcid_to_roles (line 345 in src/structure_tree.rs) inverts the hierarchy into a per-page HashMap<MCID, StructRole>. This lookup table enables O(1) resolution of semantic roles during text extraction, mapping each integer MCID to its corresponding structural label such as "H1", "P", or "Table".

Join with Text Content

The public API function extract_structure_elements_mem (defined in src/lib.rs, line 93) orchestrates the final assembly. It loads the PDF, obtains the structure tree, builds the MCID-to-role map, and returns a flat vector of StructureElement records. These records contain:

• 1-indexed page numbers
• MCID integers
• Role name strings (e.g., "H1", "TD")

Consumers then join these elements with TextItem instances from extract_text_with_positions (located in src/extractor/content_stream.rs), which also carry MCID and page metadata, producing fully-qualified semantic output.

Practical Implementation

Basic Rust Example

The following snippet demonstrates loading a tagged PDF, extracting both structural metadata and text positions, and joining them to produce semantically labeled output:

use pdf_inspector::{
    extract_text_with_positions,
    extract_structure_elements,
};

fn main() -> Result<(), pdf_inspector::PdfError> {
    // Extract raw text items including MCIDs and page numbers
    let text_items = extract_text_with_positions("document.pdf")?;
    
    // Extract semantic roles from the structure tree
    let struct_elems = extract_structure_elements("document.pdf", None)?;
    
    // Build lookup table: (page, mcid) → role
    let mut role_lookup = std::collections::HashMap::new();
    for elem in struct_elems {
        role_lookup.insert((elem.page, elem.mcid), elem.role);
    }
    
    // Attach semantic roles to extracted text
    for item in text_items {
        match role_lookup.get(&(item.page, item.mcid)) {
            Some(role) => println!("[{}] {}", role, item.text),
            None => println!("[P] {}", item.text), // Fallback paragraph
        }
    }
    
    Ok(())
}

Command-Line Usage

The shipped pdf2md binary provides immediate access to structure extraction without writing code:

pdf2md --json --extract-structure input.pdf

This outputs a JSON array of structural elements:

[
  {"page":1,"mcid":12,"role":"H1"},
  {"page":1,"mcid":34,"role":"P"},
  {"page":2,"mcid":7,"role":"Table"}
]

When combined with the standard --json text output, this data allows you to reconstruct headings with proper Markdown depth (#, ##) rather than inferring hierarchy from font sizes.

Key Source Files and Architecture

File	Purpose	Key Components
src/structure_tree.rs	Parses /StructTreeRoot and manages role mapping	StructTree::from_doc, StructTree::mcid_to_roles, StructElement, StructRole
src/lib.rs	Public API façade	extract_structure_elements_mem, extract_structure_elements, StructureElement
src/extractor/content_stream.rs	Extracts text with geometric and MCID metadata	extract_text_with_positions, TextItem
src/markdown/convert.rs	Converts extracted items to Markdown while respecting roles	Role-aware conversion pipeline
src/types.rs	Core type definitions	TextItem, PdfRect, MarkedContentRef

Summary

• pdf‑inspector extracts semantic structure by parsing the /StructTreeRoot dictionary found in tagged PDFs.
• The pipeline uses StructTree::from_doc to build the hierarchy, StructTree::mcid_to_roles to create fast lookup tables, and extract_structure_elements_mem to expose the data.
• Text items extracted via extract_text_with_positions carry MCID metadata that can be joined with structural roles to preserve headings, tables, and lists during Markdown conversion.
• The architecture gracefully handles untagged PDFs by returning empty vectors, allowing fallback to geometric-based extraction.

Frequently Asked Questions

What is the difference between tagged and untagged PDF extraction?

Tagged PDFs contain explicit semantic markup that identifies headings, paragraphs, and other logical elements regardless of visual formatting. pdf‑inspector extracts this via the structure tree to preserve document meaning. Untagged PDFs lack this metadata, forcing the library to rely solely on geometric heuristics for structure inference.

Can pdf‑inspector handle custom role mappings?

Yes. When a PDF defines a /RoleMap dictionary that maps custom role names to standard PDF structure types, StructTree::from_doc in src/structure_tree.rs resolves these mappings during tree construction. This ensures that proprietary tags are normalized to standard roles like "H1" or "Table" before reaching the output pipeline.

How do I combine structural elements with the actual text content?

Call extract_text_with_positions to obtain a vector of TextItem objects, each containing page, mcid, and text fields. Then call extract_structure_elements to get the corresponding role mappings. Join these datasets using the tuple (page, mcid) as the key, as demonstrated in the Rust example above.

What happens if I run structure extraction on an untagged PDF?

The extract_structure_elements function will return an empty vector. Because pdf‑inspector separates tree parsing from content extraction in src/lib.rs, untagged documents simply skip the semantic enhancement phase while still allowing standard text extraction to proceed normally.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}