How to Extract All HTML Chunks to Separate Files with the `--all-chunks` CLI Option

Use the betterhtmlchunking CLI with --all-chunks and --output-dir to split an HTML document into logical chunks and write each one to its own file.

The betterhtmlchunking library provides a command-line interface that processes HTML documents from stdin, identifies logical content regions, and can export each region as a standalone file. When you need to extract all chunks to separate files with the --all-chunks option, the CLI orchestrates a multi-stage pipeline defined in betterhtmlchunking/main.py and betterhtmlchunking/cli.py.

Prerequisites and Basic Syntax

The --all-chunks flag requires Python 3 and the betterhtmlchunking package installed. You must always pair --all-chunks with --output-dir (or -o) to specify where the files will be written. If the directory does not exist, the CLI creates it automatically.

cat input.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./chunks

How the --all-chunks Pipeline Works

When you enable --all-chunks, the CLI executes a four-step workflow defined in the DomRepresentation class:

1. Read HTML input from stdin and build a DOM representation, filtering unwanted tags via remove_unwanted_tags in betterhtmlchunking/utils.py.
2. Compute region-of-interest (ROI) chunks using the compute_tree_representation method in betterhtmlchunking/main.py (lines 99-149).
3. Render each ROI into HTML or plain text via the RenderSystem class, storing results in html_render_roi and text_render_roi dictionaries.
4. Write each rendered chunk to a separate file inside the --output-dir directory, as implemented in the handling block of betterhtmlchunking/cli.py (lines 134-162).

Step-by-Step CLI Examples

Basic HTML Extraction

To extract all chunks as separate HTML files, pipe your document into the CLI and specify the output directory:

cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./html_chunks

This creates files named chunk_0.html, chunk_1.html, chunk_2.html, and so on, each containing a logical content region from the original document.

Extracting Plain Text Chunks

Add the --text-only flag to output .txt files instead of HTML. This uses the text_render_roi dictionary from the render system:

cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --text-only \
    --output-dir ./text_chunks

Resulting files are named chunk_0.txt, chunk_1.txt, etc., containing only the extracted text content.

Verbose Output for Debugging

Use --verbose or --maximal-verbose to see confirmation messages as each file is written. The CLI prints Wrote ./chunks/chunk_<n>.html to stderr for each chunk:

cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./chunks \
    --verbose

Controlling Chunk Size with --max-length

Combine --all-chunks with --max-length to control the maximum size of each chunk before the splitter creates a new one:

cat page.html | python -m betterhtmlchunking.cli chunk \
    --max-length 20000 \
    --all-chunks \
    --output-dir ./small_chunks

Understanding the Output File Naming Convention

When --all-chunks is enabled, the CLI iterates over sorted ROI indices and generates filenames following the pattern chunk_<index>.<ext>:

• Index: The zero-based position of the chunk in the document sequence.
• Extension: html by default, or txt when --text-only is specified.

This logic resides in the file-writing block of betterhtmlchunking/cli.py (lines 44-53), which selects the appropriate rendered content from either html_render_roi or text_render_roi based on the --text-only flag.

Key Implementation Details

The --all-chunks functionality spans three critical files in the repository:

File	Role	Key Sections
betterhtmlchunking/cli.py	Parses CLI arguments and executes the file-writing loop.	Option definitions (lines 34-61); handling block that writes chunk files (lines 134-162).
betterhtmlchunking/main.py	Contains DomRepresentation class that builds the DOM and computes ROI chunks.	start() method running the three-step pipeline (lines 99-149).
betterhtmlchunking/render_system.py	Stores rendered outputs in html_render_roi and text_render_roi dictionaries.	Dictionaries accessed by the CLI to retrieve chunk content.

Summary

• The --all-chunks option in betterhtmlchunking extracts every logical content region from an HTML document into separate files.
• You must specify --output-dir to define where the chunk files are saved; the directory is created automatically if missing.
• Files are named chunk_<index>.html by default, or chunk_<index>.txt when using --text-only.
• The feature relies on the DomRepresentation class in betterhtmlchunking/main.py and the file-writing logic in betterhtmlchunking/cli.py.

Frequently Asked Questions

What happens if I use --all-chunks without --output-dir?

The CLI will fail because --all-chunks requires an output directory to know where to write the files. According to the argument parsing logic in betterhtmlchunking/cli.py (lines 34-61), the tool validates that --output-dir is provided when --all-chunks is enabled.

Can I control the maximum size of each chunk when using --all-chunks?

Yes, combine --all-chunks with the --max-length parameter. This sets a character limit for each chunk before the splitter creates a new one. For example: --max-length 20000 --all-chunks --output-dir ./chunks.

Does --all-chunks work with plain text output instead of HTML?

Yes, add the --text-only flag to extract plain text chunks. When enabled, the CLI writes files with .txt extensions instead of .html, using the text_render_roi dictionary from the render system rather than html_render_roi.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}