how-to-guide

How to Enable Maximal-Verbose Logging in BetterHtmlChunking to Debug Chunking Issues

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

To enable maximal-verbose logging, pass the --maximal-verbose flag when using the CLI or call set_log_level(logging.DEBUG) programmatically to activate DEBUG-level output that reveals per-node DOM statistics, ROI formation, and chunk boundaries.

The betterhtmlchunking library splits HTML documents into semantically coherent chunks, but debugging why specific nodes group together or split apart requires visibility into the internal decision pipeline. Enabling maximal-verbose mode exposes the DOM tree structure, node representation lengths, and region-of-interest (ROI) calculations that drive the chunking algorithm.

What Maximal-Verbose Logging Reveals

When you activate maximal-verbose mode, the library emits DEBUG level logs to stderr containing four critical diagnostic layers:

  • Per-node DOM statistics: Every node’s XPath, HTML length, and text length
  • ROI construction: Each region-of-interest’s total length and constituent node XPaths
  • Final chunk metrics: Character counts and boundaries for generated chunks
  • Tree state snapshots: Depth-indexed views of the DOM via log_tree_node_states

According to the source code in betterhtmlchunking/main.py (lines 13–23), the DomRepresentation.start method iterates over all nodes after tree construction and logs these details only when maximal-verbose mode is active. Similarly, betterhtmlchunking/tree_regions_system.py (lines 22–33) exposes the log_tree_node_states method that prints a detailed tree view including depths and lengths when invoked.

Enabling Debug Mode via the CLI

The simplest way to inspect DOM node details is through the command-line interface defined in betterhtmlchunking/cli.py (lines 42–44).


# Standard chunking (INFO level, minimal output)

echo "<html><body><h1>Title</h1><p>Paragraph</p></body></html>" | \
  betterhtmlchunking --max-length 1000

# Maximal-verbose: full debug dump to stderr

echo "<html><body><h1>Title</h1><p>Paragraph</p></body></html>" | \
  betterhtmlchunking --max-length 1000 --maximal-verbose

When running with --maximal-verbose, you’ll see structured debug output like:


DEBUG - betterhtmlchunking.main - Total nodes in DOM tree: 27
DEBUG - betterhtmlchunking.main - Node XPath: /html/body/h1, HTML length: 42, Text length: 11
DEBUG - betterhtmlchunking.main - Node XPath: /html/body/p[1], HTML length: 156, Text length: 89
DEBUG - betterhtmlchunking.tree_regions_system - ROI 0: HTML length 384, Nodes XPaths: ['/html/body/h1', '/html/body/p[1]', '/html/body/p[2]']
DEBUG - betterhtmlchunking.main - Generated chunk 0: HTML length 384

All diagnostic messages route to stderr, ensuring that stdout remains clean for actual chunk output, JSON results, or piped processing.

Configuring Maximal-Verbose Logging Programmatically

For Python applications integrating the chunker directly, import the logging configuration utilities from betterhtmlchunking/logging_config.py and set the level to logging.DEBUG.

import logging
from betterhtmlchunking.logging_config import setup_root_logger, set_log_level
from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.tree_regions_system import ReprLengthComparisionBy

# Initialize logger and set maximal-verbose level

setup_root_logger(level=logging.WARNING)  # baseline

set_log_level(logging.DEBUG)              # enable maximal-verbose

html_content = "<html><body><article><h1>Heading</h1><p>Content</p></article></body></html>"

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=5000,
    website_code=html_content,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

# Start processing with maximal_verbose flag to trigger internal diagnostics

dom.start(verbose=False, maximal_verbose=True)

The DomRepresentation.start method accepts a maximal_verbose boolean parameter that, when True, triggers the detailed per-node logging loop and invokes log_tree_node_states from the TreeRegionsSystem class.

Redirecting Debug Output to a File

By default, setup_root_logger (defined in betterhtmlchunking/logging_config.py, lines 26–63) attaches a StreamHandler to stderr. To capture debug logs for later analysis without cluttering the console, add a FileHandler to the root logger:

import logging
from betterhtmlchunking.logging_config import setup_root_logger

# Setup logger with DEBUG level

logger = setup_root_logger(level=logging.DEBUG)

# Create file handler for persistent logs

file_handler = logging.FileHandler("chunking-debug.log")
formatter = logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s")
file_handler.setFormatter(formatter)

# Add handler to capture all debug output

logger.addHandler(file_handler)

# Now all maximal-verbose output writes to chunking-debug.log

# while stderr still receives a copy

This configuration is particularly useful when batch-processing large HTML corpora where you need to audit chunking decisions post-hoc.

Interpreting the Debug Output

Understanding the log stream helps you diagnose specific chunking anomalies:

  • Node XPath entries reveal exactly which DOM elements the algorithm evaluates
  • HTML length vs. Text length discrepancies indicate whether inline styles, scripts, or attributes are inflating the character count
  • ROI listings show how the TreeRegionsSystem groups adjacent nodes before chunking them
  • Chunk character counts at the end of the log confirm whether chunks respect your MAX_NODE_REPR_LENGTH constraints

If you see nodes with unexpectedly large HTML lengths in the debug output, check for hidden <script> or <style> tags that might be bloating the representation without contributing visible text content.

Summary

  • Use the --maximal-verbose CLI flag to instantly enable DEBUG logging without code changes.
  • Call set_log_level(logging.DEBUG) in Python scripts to activate the same detailed output programmatically.
  • All logs write to stderr, preserving stdout for clean chunk output.
  • Key diagnostic data includes per-node XPath, HTML/text lengths, ROI compositions, and final chunk sizes.
  • Source locations: Configuration lives in betterhtmlchunking/logging_config.py, while the debug emission logic resides in betterhtmlchunking/main.py and betterhtmlchunking/tree_regions_system.py.

Frequently Asked Questions

How do I enable maximal-verbose logging only for specific chunking operations?

Set the logging level immediately before calling dom.start(), then reset it afterward. Because betterhtmlchunking uses the standard logging module, you can temporarily elevate the level for a single operation and restore it to INFO or WARNING once debugging completes, ensuring other parts of your application maintain normal log verbosity.

Does maximal-verbose logging expose sensitive HTML content in the logs?

Yes, the DEBUG output includes full XPath locations and character length measurements of your HTML nodes. If your documents contain sensitive data (passwords, PII, tokens), redirect logs to a secure file location using the FileHandler pattern shown above, or sanitize the HTML before processing to prevent credential leakage in log files.

Why are my logs going to stderr instead of stdout?

The setup_root_logger function in betterhtmlchunking/logging_config.py explicitly attaches a StreamHandler to sys.stderr (lines 26–63). This design keeps diagnostic noise separate from your primary output stream, allowing you to pipe clean JSON or HTML chunks to other tools while still capturing debug information for troubleshooting.

What is the difference between the verbose and maximal_verbose parameters in DomRepresentation.start()?

The verbose parameter controls basic progress indicators, while maximal_verbose triggers the deep DOM inspection via log_tree_node_states. Setting maximal_verbose=True in betterhtmlchunking/main.py (line 13) activates the per-node logging loop and ROI dumps, whereas verbose=True only toggles high-level status messages without the granular node details.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →