Using MarkdownProfile (Fidelity vs Compact) for Token-Efficient Output in pdf-inspector

The pdf-inspector library provides two built-in Markdown profiles—Fidelity for exact layout preservation and Compact for aggressive token reduction—controlled via the MarkdownProfile enum at the CLI or programmatically.

pdf-inspector converts PDF documents to Markdown with a configurable trade-off between visual accuracy and token economy. The MarkdownProfile enum in src/markdown/mod.rs defines this behavior, allowing developers to optimize output for either human readability or LLM consumption.

What Is MarkdownProfile?

MarkdownProfile is a Rust enum that determines how aggressively the post-processing pipeline normalizes whitespace and cosmetic formatting:

pub enum MarkdownProfile { Fidelity, Compact }

• Fidelity: Preserves exact line breaks, spaces, and dot-leader characters (e.g., "…..")
• Compact: Collapses multiple spaces, removes dot-leaders, and trims trailing whitespace

This enum is declared at line 913 of src/markdown/mod.rs and propagates through the entire extraction pipeline.

How Each Profile Transforms Output

The profile selection affects three distinct stages:

Stage	Fidelity Behavior	Compact Behavior
Extraction	Keeps exact spacing and dot-leaders	Normalizes whitespace, strips dot-leaders
Post-processing (src/markdown/postprocess.rs)	fidelity_profile_preserves_dot_leaders leaves formatting intact	compact_profile_collapses_dot_leaders removes/condenses decorative characters
Final output	Verbatim structure for tables and indentation	Streamlined text optimized for token count

The post-processing functions are implemented starting at line 10 of src/markdown/postprocess.rs.

When to Use Each Profile

Choose Fidelity when:

• Downstream tools require precise visual structure (tables, nested lists, code blocks)
• Human readers need preserved indentation and alignment
• Token budget is not constrained

Choose Compact when:

• Feeding output directly to LLM prompts or embeddings
• Processing large PDFs where every token counts
• Running batch pipelines across thousands of documents

Compact mode can reduce token count by approximately 30% without sacrificing semantic content. This makes it ideal for RAG applications and automated document analysis workflows.

CLI Usage

The pdf2md binary defaults to Fidelity. Enable Compact with the --compact flag:

# Full-fidelity output (default)

pdf2md my-document.pdf > output.md

# Compact, token-efficient output

pdf2md --compact my-document.pdf > output_compact.md

The flag parsing occurs at line 312 of src/bin/pdf2md.rs, where --compact maps to MarkdownProfile::Compact.

Programmatic Usage (Rust)

Configure the profile explicitly in PdfOptions:

use pdf_inspector::{process_pdf_with_options, MarkdownProfile};

let opts = pdf_inspector::PdfOptions {
    markdown: pdf_inspector::MarkdownOptions {
        profile: MarkdownProfile::Compact,
        ..Default::default()
    },
    ..Default::default()
};

let result = process_pdf_with_options("my-document.pdf", opts)?;
println!("{}", result.markdown.text);

Python Binding

Via the pyo3 interface:

import pdf_inspector

opts = pdf_inspector.PdfOptions()
opts.markdown.profile = pdf_inspector.MarkdownProfile.Compact

result = pdf_inspector.process_pdf("my-document.pdf", opts)
print(result.markdown.text)

Key Source Files

File	Purpose
src/markdown/mod.rs	MarkdownProfile enum definition (line 913)
src/markdown/postprocess.rs	Profile-specific dot-leader handling (lines 10–12)
src/bin/pdf2md.rs	CLI flag parsing for --compact (line 312)
tests/integration_tests.rs	Verification tests for both profiles (lines 537–563)

Testing Profile Behavior

The integration test suite validates both profiles:

• Lines 537+: Fidelity profile produces full expected output with preserved formatting
• Lines 547+: Compact profile demonstrates measurable token reduction while maintaining semantic equivalence

These tests ensure that profile selection behaves predictably across PDF document types.

Summary

• MarkdownProfile controls the fidelity-versus-efficiency trade-off in pdf-inspector
• Fidelity preserves exact visual structure; Compact minimizes tokens through aggressive normalization
• Configure via CLI --compact flag or MarkdownOptions.profile in Rust/Python
• Compact reduces token count ~30%—critical for LLM pipelines
• Core implementation spans mod.rs, postprocess.rs, and pdf2md.rs

Frequently Asked Questions

How do I enable Compact mode from the command line?

Pass the --compact flag to pdf2md. The default behavior without this flag uses Fidelity mode, as implemented in src/bin/pdf2md.rs at line 312.

Will Compact mode break table formatting?

Compact mode removes cosmetic whitespace and dot-leaders but preserves semantic structure. Tables remain parseable as Markdown, though visual alignment of cell content may shift. For applications requiring pixel-perfect table reproduction, use Fidelity mode.

Can I switch profiles dynamically at runtime?

Yes. The MarkdownProfile is passed through PdfOptions to process_pdf_with_options, allowing runtime selection based on document type, user preference, or downstream consumer requirements.

What exactly does Compact mode remove?

According to src/markdown/postprocess.rs, Compact mode collapses multiple consecutive spaces, strips dot-leader sequences (repeated periods used for visual alignment), and trims trailing whitespace from lines. These changes target token-heavy formatting that carries no semantic meaning for language models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}