How to Debug PDF Classification with RUST_LOG in pdf-inspector

Set the RUST_LOG environment variable to enable structured debug output for any module in the pdf-inspector Rust codebase, with logs written to stderr to preserve normal program output.

The pdf-inspector repository from Firecrawl uses the log crate combined with env_logger to provide deep visibility into its PDF classification pipeline. When you need to understand why a page was classified as TextBased, Scanned, ImageBased, or Mixed, structured logging lets you inspect the intermediate values driving every decision.

How RUST_LOG Works in pdf-inspector

The logging system follows the standard env_logger pattern: module_path=level. Each major component contains log::debug! and log::trace! statements that report operator counts, font metadata, layout heuristics, and detection thresholds.

Key modules and their paths:

• pdf_inspector::detector — core classification logic in src/detector.rs
• pdf_inspector::extractor::layout — column detection and reading-order logic in src/extractor/layout.rs
• pdf_inspector::tables — table detection strategies in src/tables/mod.rs
• pdf_inspector::markdown::analysis — paragraph threshold calculations in src/markdown/analysis.rs
• pdf_inspector::extractor::content_stream — raw content-stream operator analysis
• pdf_inspector::extractor::fonts — font metadata and CMap parsing

Logs emit to stderr, so piping stdout to /dev/null or a file keeps your Markdown or JSON output clean.

Common RUST_LOG Patterns for PDF Classification Debugging

Inspect the PDF Type Detector

To see why a document was classified as scanned or mixed:

RUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf

This activates debug statements in src/detector.rs that print per-page analysis values.

Debug Content-Stream Operators

For the lowest-level view of operator counts that determine if a page has extractable text:

RUST_LOG=pdf_inspector::extractor::content_stream=trace cargo run --bin pdf2md -- file.pdf > /dev/null

Use trace level for maximum verbosity on text operations.

Analyze Font and Encoding Issues

When characters decode incorrectly or ligatures fail:

RUST_LOG=pdf_inspector::extractor::fonts=debug cargo run --bin pdf2md -- file.pdf > /dev/null

This reveals CMap parsing decisions and ToUnicode mapping problems in the font extraction pipeline.

Examine Column and Reading-Order Logic

For newspaper-style layouts or multi-column documents:

RUST_LOG=pdf_inspector::extractor::layout=debug cargo run --bin pdf2md -- file.pdf > /dev/null

The src/extractor/layout.rs module logs histogram valleys and derived column counts.

Trace Table Detection

To follow the three detection strategies (rect-based, line-based, heuristic):

RUST_LOG=pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf > /dev/null

Monitor Paragraph Threshold Analysis

For debugging Y-gap heuristics that segment text into paragraphs:

RUST_LOG=pdf_inspector::markdown::analysis=debug cargo run --bin pdf2md -- file.pdf > /dev/null

Enable All Debug Output

For comprehensive tracing across all modules:

RUST_LOG=pdf_inspector=debug cargo run --bin pdf2md -- file.pdf > /dev/null

Combine Multiple Modules

Use comma-separated filters for focused cross-module debugging:

RUST_LOG=pdf_inspector::detector=debug,pdf_inspector::tables=debug cargo run --bin pdf2md -- file.pdf

Interpreting Debug Output

When detector logging is active, you'll see structured entries like:

DEBUG pdf_inspector::detector::detect_from_document: page 3: text_ops=12 images=1 image_count=1 template=0 unique_chars=27 alphanum=5 path_ops=0 vector_text=false image_area=0 identity_h_no_tounicode=false type3_only=false font_changes=3 decodable_fonts=true

These fields map directly to the PageAnalysis struct in src/detector.rs. Use them to answer specific classification questions:

• Why scanned? — Check text_ops below threshold, images >= 1, and low alphanum count
• Why mixed requiring OCR? — Look for has_template_image or has_vector_text flags
• Why wrong column count? — Check pdf_inspector::extractor::layout logs for valley detection and early-exit conditions

Filtering Output with Grep

Pipe stderr to narrow results:

RUST_LOG=pdf_inspector::detector=debug cargo run --release --bin detect-pdf -- file.pdf 2>&1 | grep "pages with text"

This pattern isolates specific metrics without drowning in trace-level noise.

Key Source Files for Debugging

File	Module Path	Purpose
src/detector.rs	pdf_inspector::detector	PDF type detection and OCR recommendations
src/extractor/layout.rs	pdf_inspector::extractor::layout	Column detection and reading-order heuristics
src/tables/mod.rs	pdf_inspector::tables	Table detection pipeline
src/markdown/analysis.rs	pdf_inspector::markdown::analysis	Paragraph-gap analysis
src/tounicode.rs	pdf_inspector::tounicode	CMap and ToUnicode parsing logs

For quick reference, see docs/debugging.md and AGENTS.md in the repository root.

Summary

• Set RUST_LOG with module path and level to activate specific debug output
• Use debug for operational metrics, trace for exhaustive operator-level detail
• Pipe stdout to isolate logs when generating Markdown or JSON output
• Read field names in log output against PageAnalysis in src/detector.rs to trace classification decisions
• Combine modules with comma-separated filters for cross-cutting investigations

Frequently Asked Questions

What log levels does pdf-inspector support?

The env_logger backend recognizes error, warn, info, debug, and trace. For PDF classification debugging, use debug for summary metrics and trace for raw operator counts. The default level is error when RUST_LOG is unset.

How do I debug why a page was incorrectly marked as scanned?

Set RUST_LOG=pdf_inspector::detector=debug and examine the text_ops, images, and alphanum fields. A page becomes Scanned when text_ops falls below min_text_ops_per_page (default 3) while containing images with insufficient alphanumeric characters.

Can I log to a file instead of stderr?

Redirect stderr in your shell: RUST_LOG=pdf_inspector=debug cargo run ... 2> debug.log. The env_logger configuration in pdf-inspector does not include file appenders, so shell redirection is the standard approach.

Where are the threshold constants defined?

Detection thresholds live in src/detector.rs as const values within the detection logic. Debug output shows the computed values compared against these thresholds, letting you identify which constraint triggered a classification without reading source code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}