How to Debug PDF Extraction Layout with RUST_LOG in firecrawl/pdf-inspector

Set the RUST_LOG environment variable to pdf_inspector::extractor::layout=debug to emit diagnostic messages from the layout detection pipeline to stderr, allowing you to trace column detection, validation failures, and fallback paths in real-time.

The firecrawl/pdf-inspector repository uses the standard Rust log crate for structured diagnostics throughout its PDF-to-Markdown extraction pipeline. When dealing with complex multi-column documents, debugging the layout detection logic is essential for resolving reading-order errors or misidentified column boundaries. By configuring the RUST_LOG environment variable, you can inspect the internal state of the histogram-based column detection algorithm without modifying source code.

Core Layout Detection Functions

The layout detection logic resides in src/extractor/layout.rs. This module implements a projection-profile algorithm that identifies column gutters by analyzing horizontal whitespace valleys. Three primary functions orchestrate the detection process, each emitting structured debug logs when RUST_LOG is configured appropriately.

The detect_columns Entry Point

The detect_columns function (lines 21–38) serves as the primary entry point for column analysis. It builds a horizontal projection histogram of text items, identifies valleys (gutters) in the distribution, and constructs ColumnRegion structures. When debugging is enabled, this function logs the page number and item count at initialization, allowing you to verify that the correct TextItem set is entering the layout engine.

The try_xy_cut_split Fallback

When primary valley detection fails, the algorithm falls back to try_xy_cut_split (lines 138–166). This function searches for the largest horizontal gap between items and attempts to split the page into two columns. Debug output from this function includes the gap coordinate (x=…), gap width in points, and the resulting left/right item distributions, helping you identify why the primary detection missed a column boundary.

The validate_and_build_columns Validator

The validate_and_build_columns function (lines 225–260) filters candidate valleys against heuristics. It checks vertical overlap between potential columns, excludes list-marker artifacts, and enforces minimum item thresholds (MIN_ITEMS_MAJOR and MIN_ITEMS_MINOR). Debug messages indicate whether valleys passed validation or if all candidates were rejected, triggering additional fallback mechanisms.

Interpreting RUST_LOG Debug Output

When you set RUST_LOG=pdf_inspector::extractor::layout=debug, the library streams diagnostic messages to stderr. These logs trace the execution flow through the detection pipeline:

• page X: detect_columns: N items — Indicates the start of analysis for a specific page and the cardinality of the input set.
• page X: XY-cut split at x=… (gap=…pt, left=…, right=…) — Emitted by try_xy_cut_split when the fallback path activates.
• page X: relative valley detection found N columns — Logs results from alternative valley detection strategies.
• page X: … valleys found but none passed validation — Indicates validate_and_build_columns rejected all candidates due to heuristic failures.
• page X: … columns detected (boundaries: …) — Confirms successful detection with final boundary coordinates.

Debugging Specific Layout Issues

Tracing these log messages allows you to diagnose five critical aspects of the extraction process:

Item Collection Verification. Confirm that the expected TextItem objects are reaching detect_columns. If the item count differs from the visible text blocks on the page, the issue likely resides in the upstream content stream parser rather than the layout module.

Histogram Threshold Analysis. The algorithm uses constants like NOISE_FRACTION and MIN_GUTTER_WIDTH to filter histogram noise. Debug output reveals the calculated thresholds versus actual valley depths, helping you determine if whitespace gaps are being incorrectly classified as noise.

Valley Geometry Inspection. Review the gutter positions relative to page margins. The debug logs show whether detected valleys meet the minimum distance requirements from page edges and whether they align with visual column boundaries.

Validation Heuristic Evaluation. When columns are detected but rejected, the logs indicate which MIN_ITEMS constraint or vertical overlap check failed. This is particularly useful for documents with narrow sidebars or table-of-contents columns that might not meet the major/minor item ratio requirements.

Fallback Path Tracing. The logs reveal whether the algorithm utilized the XY-cut method or relative valley detection after primary detection failed, helping you understand why certain layouts trigger alternative parsing strategies.

Command Examples for Debugging

Use these shell commands to enable diagnostics in different scenarios:

# Enable layout debugging only, suppressing normal output

RUST_LOG=pdf_inspector::extractor::layout=debug \
cargo run --bin pdf2md -- example.pdf > /dev/null

# Debug layout alongside table detection for documents with tabular columns

RUST_LOG=pdf_inspector::extractor::layout=debug,pdf_inspector::tables=debug \
cargo run --bin pdf2md -- example.pdf > /dev/null

# Capture debug output to a file for offline analysis

RUST_LOG=pdf_inspector::extractor::layout=debug \
cargo run --bin pdf2md -- example.pdf 2> layout.log

# Correlate raw PDF operators with layout decisions

RUST_LOG=pdf_inspector::extractor::content_stream=trace,pdf_inspector::extractor::layout=debug \
cargo run --bin pdf2md -- example.pdf > /dev/null

Command Component Breakdown:

• RUST_LOG=pdf_inspector::extractor::layout=debug — Targets debug-level logs from the layout module specifically.
• > /dev/null — Discards the generated Markdown output (stdout), leaving only diagnostic logs (stderr) visible.
• 2> layout.log — Redirects stderr to a file, preserving the debug stream for analysis in text editors or grep pipelines.
• content_stream=trace — Enables fine-grained tracing of PDF operator parsing in src/extractor/content_stream.rs, useful when layout issues stem from incorrect text item generation.

Related Debugging Modules

While src/extractor/layout.rs handles column detection, several adjacent modules influence layout behavior and provide complementary debug information:

src/extractor/content_stream.rs — Controls the low-level PDF operator parser. When layout detection appears to miss text blocks, enable pdf_inspector::extractor::content_stream=trace to verify that text positioning operators are being interpreted correctly before they reach the layout engine.

src/tables/mod.rs — Implements table detection logic that executes after column identification. If columns contain embedded tables, debugging both modules simultaneously (layout=debug,tables=debug) reveals whether table structures are interfering with column boundary detection.

docs/debugging.md — Contains the official reference for all available RUST_LOG targets and verbosity levels within the repository.

src/markdown/analysis.rs — Logs Y-gap and paragraph-threshold analysis that follows layout detection. Use this module's debug output to verify that correctly detected columns are being rendered with proper reading order in the final Markdown output.

Summary

• The firecrawl/pdf-inspector library uses the Rust log crate to emit diagnostic information when the RUST_LOG environment variable is set.
• Layout-specific debugging requires RUST_LOG=pdf_inspector::extractor::layout=debug to trace the detect_columns, try_xy_cut_split, and validate_and_build_columns functions in src/extractor/layout.rs.
• Debug output streams to stderr and includes histogram thresholds, valley coordinates, validation results, and fallback activation signals.
• Combine layout debugging with content_stream=trace or tables=debug to diagnose interactions between text parsing, column detection, and table extraction.
• Redirect stderr to log files (2> layout.log) for offline analysis of complex multi-page documents.

Frequently Asked Questions

What environment variable controls logging in pdf-inspector?

The repository follows the Rust ecosystem standard by respecting the RUST_LOG environment variable. This variable accepts module paths and log levels (e.g., debug, trace, info) to filter diagnostic output from specific components like pdf_inspector::extractor::layout.

How do I capture debug output to a file instead of the terminal?

Since debug logs write to stderr, redirect stream 2 to a file using shell redirection. Execute your command with 2> layout.log appended to save all RUST_LOG output to layout.log while keeping the normal program output (Markdown text) in the terminal or redirecting it elsewhere.

Why am I not seeing any layout debug messages even with RUST_LOG set?

Verify that you are targeting the correct module path. The layout module specifically requires pdf_inspector::extractor::layout=debug (note the double colon separators and the underscore in pdf_inspector). Additionally, ensure you are not suppressing stderr with shell redirection, and confirm that the binary was compiled without --release flags that might strip debug symbols or log statements in some configurations.

How can I correlate low-level PDF operators with layout decisions?

Enable tracing for both the content stream parser and the layout module simultaneously using RUST_LOG=pdf_inspector::extractor::content_stream=trace,pdf_inspector::extractor::layout=debug. This configuration prints the raw PDF operators (from src/extractor/content_stream.rs) alongside the layout detection logs, allowing you to map specific text positioning commands to the resulting column boundaries detected in layout.rs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}