How pdf-inspector Extracts MCID References from Tagged PDFs via Structure Tree Parsing

pdf-inspector extracts MCID references by traversing the PDF's Structure Tree to build per-page lookup tables mapping Marked-Content IDs to semantic roles, then correlating these MCIDs with content stream operators to preserve document hierarchy in the final output.

The open-source pdf-inspector project provides a robust Rust implementation for parsing tagged PDFs. At the heart of its architecture lies a three-stage pipeline that bridges the gap between low-level PDF content streams and high-level semantic structure. This article examines how pdf-inspector's structure tree parsing extracts Marked-Content IDs (MCIDs)—the critical link between structural elements and page content.

The Role of MCIDs in Tagged PDFs

Tagged PDFs contain a Structure Tree that defines semantic roles like headings, paragraphs, and tables. Each structural element is associated with one or more MCIDs, which appear as markers in the content stream. MCID extraction is therefore essential for mapping raw text operators back to their intended document semantics.

The challenge lies in the PDF specification's flexibility: MCIDs can appear as dictionary entries, bare integers in /K arrays, or nested within MCR (Marked Content Reference) dictionaries. pdf-inspector handles all three cases through systematic tree traversal.

Stage 1: Structure Tree Parsing in src/structure_tree.rs

The entry point StructTree::from_doc in src/structure_tree.rs receives a lopdf::Document and builds the complete MCID-to-role mapping infrastructure.

Building the Role Map

Before traversing the tree, pdf-inspector normalizes role names:

• parse_role_map resolves custom role names to standard PDF-2.0 roles (e.g., "H1" → StructRole::Heading(1))
• This ensures consistent semantic interpretation regardless of PDF authoring tool

Core Recursive Traversal

The traversal follows a strict hierarchy through parse_kids → parse_kid → parse_struct_element_dict:

// Conceptual flow from src/structure_tree.rs
// parse_kids iterates over /K array entries
// parse_kid dispatches based on entry type
// parse_struct_element_dict handles element dictionaries

For each structural element, the code at approximately lines 770–785 performs MCID detection:

• Checks dict.get(b"MCID") for direct MCID entries
• Wraps bare integers in /K arrays as content-reference children
• Stores results in FlatStructElement objects containing MCID, page number, and resolved role

The collect_mcids_recursive function (line ≈488) gathers all MCIDs recursively, building mcid_to_roles: a HashMap<usize, HashMap<i32, StructRole>> keyed by page number.

Handling Edge Cases

pdf-inspector's structure tree parsing is robust to several PDF variations:

Pattern	Handling
Direct MCID entry	Extracted via dict.get(b"MCID")
Bare integer in /K	Wrapped as content-reference child
MCR dictionary	Parsed via dedicated MCR handling logic
Aliased roles	Resolved through role map normalization

The budget-driven parsing logic (lines ≈1850–1900) prevents runaway recursion on malformed documents.

Stage 2: Content Stream Correlation in src/extractor/content_stream.rs

With the MCID-to-role map established, pdf-inspector processes content streams to attach structural metadata to extracted text.

The Marked-Content Stack

The extractor maintains a marked-content stack during operator streaming:

// From src/extractor/content_stream.rs
// Tracks nested BMC/EMC and BDC/EMC operators
// get_innermost_mcid retrieves the current active MCID

As operators execute—particularly text-showing operators Tj and TJ—the current MCID (if any) is captured and attached to each TextItem. This produces a stream of TextItem objects annotated with their originating MCID.

Stage 3: Role Assignment in src/markdown/convert.rs

The final stage resolves semantic roles for output generation. The function resolve_dominant_role (line ≈474) performs this mapping:

1. Looks up each TextItem's MCID in the per-page map built during Stage 1
2. Aggregates roles across text items forming a line
3. Assigns the most frequent role as the line's semantic classification

This dominant role resolution determines whether a line becomes a header, list item, code block, or paragraph in the Markdown output.

Practical Usage Example

use pdf_inspector::process_pdf_with_options;
use pdf_inspector::process_mode::ProcessOptions;

// Configure extraction to include structure tree parsing
let opts = ProcessOptions::default()
    .with_structure_tree(true);

// Execute MCID extraction pipeline
let result = process_pdf_with_options("tagged_document.pdf", opts)
    .expect("PDF processing failed");

// Access the per-page MCID → role mapping
let mcid_map = result.structure_tree.unwrap();

// Query specific MCID on page 3
let page_3_roles = mcid_map.get(&3).unwrap();
let role = page_3_roles.get(&42).unwrap();

// Output includes semantic structure
println!("MCID 42 is: {:?}", role); // StructRole::Heading(2)

// Full JSON output with MCID annotations
let json_output = result.to_json().unwrap();

Key Implementation Files

File	Responsibility in MCID Extraction
src/structure_tree.rs	Parses Structure Tree, builds MCID→role maps, handles MCR dictionaries and bare MCIDs
src/extractor/content_stream.rs	Streams operators, tracks MCID context, annotates TextItems
src/markdown/convert.rs	Resolves dominant roles per line using MCID lookup
src/lib.rs	Public API (process_pdf_with_options) exposing the complete pipeline

Summary

pdf-inspector extracts MCID references from tagged PDFs through a coordinated three-stage process:

• Structure tree traversal builds per-page MCID-to-role lookup tables with robust handling of bare MCIDs and MCR dictionaries
• Content stream correlation attaches MCID context to extracted text items via a marked-content stack
• Role resolution maps MCIDs back to semantic roles for structured Markdown/JSON output

This architecture preserves the original PDF's semantic hierarchy while remaining resilient to specification variations and malformed documents.

Frequently Asked Questions

What is an MCID in a tagged PDF?

An MCID (Marked-Content ID) is an integer identifier that links structural elements in the PDF's Structure Tree to specific locations in the content stream. When a PDF reader encounters an MCID marker during content rendering, it can associate that content with its semantic role—such as heading level, table cell, or list item—defined in the Structure Tree.

How does pdf-inspector handle malformed or non-standard tagged PDFs?

pdf-inspector employs budget-driven parsing logic (lines ≈1850–1900 in src/structure_tree.rs) to prevent infinite recursion on circular structure references. It also normalizes custom role names through parse_role_map and gracefully handles three MCID representation patterns: direct dictionary entries, bare integers in /K arrays, and full MCR dictionaries. These defensive measures allow successful extraction from PDFs generated by diverse authoring tools.

Can I extract MCID mappings without generating Markdown output?

Yes. The ProcessOptions::with_structure_tree(true) flag enables structure tree parsing independently of output format. The resulting HashMap<usize, HashMap<i32, StructRole>> from result.structure_tree provides raw MCID-to-role mappings for custom processing pipelines. You can also access the complete extraction result via result.to_json() which includes MCID annotations alongside text content.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}