betterhtmlchunking CLI Logging Modes: Default vs Verbose vs Maximal-Verbose

The betterhtmlchunking CLI provides three distinct logging levels—default (WARNING), --verbose (INFO), and --maximal-verbose (DEBUG)—that determine whether you see only critical errors, high-level pipeline progress, or exhaustive internal diagnostics during HTML chunking operations.

The carlosplanchon/betterhtmlchunking library uses Python’s standard logging module to provide visibility into its three-step HTML chunking pipeline. Understanding the differences between the default, verbose, and maximal-verbose logging modes helps you debug issues or monitor progress without overwhelming your terminal output.

How Logging Works in betterhtmlchunking

Root Logger Configuration

In betterhtmlchunking/logging_config.py, the function setup_root_logger() initializes the logging infrastructure with a default level of WARNING (lines 26-33). This configuration installs a console handler that writes to stderr, ensuring that log output never interferes with the actual chunked HTML content sent to stdout. The logger’s propagate flag is disabled, restricting output to the package’s named logger hierarchy (betterhtmlchunking.<module>).

CLI Argument Processing

The betterhtmlchunking/cli.py file handles command-line interface logic using Typer. After parsing arguments, the CLI adjusts the logger level via set_log_level(), which calls logger.setLevel(level) based on the flag hierarchy: --maximal-verbose takes precedence, followed by --verbose, with no flags leaving the default WARNING level intact.

Default vs Verbose vs Maximal-Verbose Logging Modes

Default Mode (WARNING)

When running without flags, the CLI operates in default mode with the logger set to WARNING. You will only see warnings or errors emitted by the library—such as invalid arguments or processing failures. No pipeline-step messages appear, making this mode ideal for production scripts where silent operation is preferred.

Verbose Mode (INFO)

Activating --verbose (or -v) sets the logger to INFO, revealing high-level pipeline progress. In betterhtmlchunking/main.py, the DomRepresentation.start() method emits logger.info calls for each major step: "Starting DOM representation processing", "Step 1/3: Computing tree representation", "Step 2/3: Computing tree regions system", "Step 3/3: Computing render", and "Processing complete: Generated X chunks". This mode provides human-readable progress tracking without excessive detail.

Maximal-Verbose Mode (DEBUG)

Using --maximal-verbose sets the logger to DEBUG, displaying all INFO messages plus exhaustive internal diagnostics. Throughout betterhtmlchunking/main.py (lines 13-45) and tree-processing modules, logger.debug statements output node counts, per-node XPath expressions, HTML/text length details, ROI (region-of-interest) summaries, and per-chunk size information. This developer-oriented mode is essential for debugging complex HTML inputs or understanding internal chunking decisions.

Code Examples: Using Each Logging Mode

The following examples demonstrate the three logging modes when chunking an HTML file with a 40,000-character limit:

# Default mode: silent unless errors occur

cat page.html | betterhtmlchunking chunk -l 40000 > chunk_0.html

# Verbose mode: high-level pipeline progress

cat page.html | betterhtmlchunking chunk -v -l 40000

# Output to stderr:

# INFO - betterhtmlchunking.main - Starting DOM representation processing

# INFO - betterhtmlchunking.main - Step 1/3: Computing tree representation

# INFO - betterhtmlchunking.main - Step 2/3: Computing tree regions system

# INFO - betterhtmlchunking.main - Step 3/3: Computing render

# INFO - betterhtmlchunking.main - Processing complete: Generated 7 chunks

# Maximal-verbose mode: detailed diagnostics

cat page.html | betterhtmlchunking chunk --maximal-verbose -l 40000

# Sample debug output to stderr:

# DEBUG - betterhtmlchunking.main - Total nodes in DOM tree: 1523

# DEBUG - betterhtmlchunking.main - Node XPath: /html/body/div[1] HTML length: 342 Text length: 210

# DEBUG - betterhtmlchunking.main - Total ROIs (chunks): 7

# DEBUG - betterhtmlchunking.main - ROI 0: HTML length 3987 Nodes XPaths: [...]

# DEBUG - betterhtmlchunking.main - Chunk 0: HTML 3987 chars, Text 2145 chars

Note: All log messages are written to stderr, while the chunked HTML content is written to stdout (or to files when using --all-chunks).

Key Source Files

Understanding these files helps clarify how the logging modes are implemented:

• betterhtmlchunking/logging_config.py – Contains setup_root_logger() and set_log_level() functions that configure the root logger with a default WARNING level and manage runtime level changes.
• betterhtmlchunking/cli.py – Implements the Typer-based command-line interface, parsing --verbose and --maximal-verbose flags and invoking set_log_level() accordingly.
• betterhtmlchunking/main.py – Houses the DomRepresentation.start() method and the majority of logger.info and logger.debug calls that produce the observable output differences between modes.

Summary

• Default mode operates at the WARNING level, showing only errors and warnings to stderr while keeping stdout clean for chunked output.
• Verbose mode (-v or --verbose) elevates logging to INFO, displaying high-level pipeline progress messages from betterhtmlchunking/main.py during the three-step chunking process.
• Maximal-verbose mode (--maximal-verbose) sets the level to DEBUG, revealing exhaustive internal diagnostics including node counts, XPath details, and per-chunk statistics throughout the codebase.
• All modes write logs to stderr and chunked HTML to stdout, ensuring that logging never corrupts output data.

Frequently Asked Questions

What is the difference between verbose and maximal-verbose in betterhtmlchunking?

Verbose mode (--verbose or -v) sets the logger to INFO and displays high-level progress messages such as "Step 1/3: Computing tree representation" and "Processing complete: Generated X chunks". Maximal-verbose mode (--maximal-verbose) sets the logger to DEBUG and shows all INFO messages plus detailed diagnostics including DOM node counts, XPath expressions for individual nodes, and per-chunk HTML/text length statistics.

Does enabling verbose logging affect the chunked HTML output?

No. All log messages are written to stderr, while the chunked HTML content is written to stdout (or to files when using --all-chunks). This separation ensures that enabling --verbose or --maximal-verbose never corrupts or pollutes your output data, making these flags safe to use in production pipelines.

Which logging mode should I use for debugging complex HTML documents?

Use maximal-verbose mode (--maximal-verbose) when debugging complex HTML documents. This mode activates DEBUG level logging throughout betterhtmlchunking/main.py and the tree-processing modules, revealing internal state such as total node counts, per-node XPath details, ROI (region-of-interest) summaries, and exact character counts for each chunk. This information is essential for understanding why specific chunk boundaries were chosen.

Where are the logging levels configured in the source code?

The logging infrastructure is configured in three key files: betterhtmlchunking/logging_config.py defines setup_root_logger() with a default WARNING level and provides set_log_level() for runtime adjustments; betterhtmlchunking/cli.py parses the --verbose and --maximal-verbose flags and invokes set_log_level() accordingly; and betterhtmlchunking/main.py contains the actual logger.info and logger.debug calls that produce the observable output differences between modes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}