# How to Extract All HTML Chunks to Separate Files with the `--all-chunks` CLI Option

> Easily split HTML into separate files using betterhtmlchunking CLI with the --all-chunks option. Learn how to extract all HTML chunks efficiently for better organization.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Use the `betterhtmlchunking` CLI with `--all-chunks` and `--output-dir` to split an HTML document into logical chunks and write each one to its own file.**

The `betterhtmlchunking` library provides a command-line interface that processes HTML documents from `stdin`, identifies logical content regions, and can export each region as a standalone file. When you need to extract all chunks to separate files with the `--all-chunks` option, the CLI orchestrates a multi-stage pipeline defined in [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py) and [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py).

## Prerequisites and Basic Syntax

The `--all-chunks` flag requires Python 3 and the `betterhtmlchunking` package installed. You must always pair `--all-chunks` with `--output-dir` (or `-o`) to specify where the files will be written. If the directory does not exist, the CLI creates it automatically.

```bash
cat input.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./chunks

```

## How the `--all-chunks` Pipeline Works

When you enable `--all-chunks`, the CLI executes a four-step workflow defined in the `DomRepresentation` class:

1. **Read HTML input** from `stdin` and build a DOM representation, filtering unwanted tags via `remove_unwanted_tags` in [`betterhtmlchunking/utils.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/utils.py).
2. **Compute region-of-interest (ROI) chunks** using the `compute_tree_representation` method in [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py) (lines 99-149).
3. **Render each ROI** into HTML or plain text via the `RenderSystem` class, storing results in `html_render_roi` and `text_render_roi` dictionaries.
4. **Write each rendered chunk** to a separate file inside the `--output-dir` directory, as implemented in the handling block of [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 134-162).

## Step-by-Step CLI Examples

### Basic HTML Extraction

To extract all chunks as separate HTML files, pipe your document into the CLI and specify the output directory:

```bash
cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./html_chunks

```

This creates files named [`chunk_0.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_0.html), [`chunk_1.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_1.html), [`chunk_2.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_2.html), and so on, each containing a logical content region from the original document.

### Extracting Plain Text Chunks

Add the `--text-only` flag to output `.txt` files instead of HTML. This uses the `text_render_roi` dictionary from the render system:

```bash
cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --text-only \
    --output-dir ./text_chunks

```

Resulting files are named [`chunk_0.txt`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_0.txt), [`chunk_1.txt`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_1.txt), etc., containing only the extracted text content.

### Verbose Output for Debugging

Use `--verbose` or `--maximal-verbose` to see confirmation messages as each file is written. The CLI prints `Wrote ./chunks/chunk_<n>.html` to **stderr** for each chunk:

```bash
cat page.html | python -m betterhtmlchunking.cli chunk \
    --all-chunks \
    --output-dir ./chunks \
    --verbose

```

### Controlling Chunk Size with `--max-length`

Combine `--all-chunks` with `--max-length` to control the maximum size of each chunk before the splitter creates a new one:

```bash
cat page.html | python -m betterhtmlchunking.cli chunk \
    --max-length 20000 \
    --all-chunks \
    --output-dir ./small_chunks

```

## Understanding the Output File Naming Convention

When `--all-chunks` is enabled, the CLI iterates over sorted ROI indices and generates filenames following the pattern `chunk_<index>.<ext>`:

- **Index**: The zero-based position of the chunk in the document sequence.
- **Extension**: `html` by default, or `txt` when `--text-only` is specified.

This logic resides in the file-writing block of [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 44-53), which selects the appropriate rendered content from either `html_render_roi` or `text_render_roi` based on the `--text-only` flag.

## Key Implementation Details

The `--all-chunks` functionality spans three critical files in the repository:

| File | Role | Key Sections |
|------|------|--------------|
| **[`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py)** | Parses CLI arguments and executes the file-writing loop. | Option definitions (lines 34-61); handling block that writes chunk files (lines 134-162). |
| **[`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py)** | Contains `DomRepresentation` class that builds the DOM and computes ROI chunks. | `start()` method running the three-step pipeline (lines 99-149). |
| **[`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py)** | Stores rendered outputs in `html_render_roi` and `text_render_roi` dictionaries. | Dictionaries accessed by the CLI to retrieve chunk content. |

## Summary

- The `--all-chunks` option in `betterhtmlchunking` extracts every logical content region from an HTML document into separate files.
- You must specify `--output-dir` to define where the chunk files are saved; the directory is created automatically if missing.
- Files are named `chunk_<index>.html` by default, or `chunk_<index>.txt` when using `--text-only`.
- The feature relies on the `DomRepresentation` class in [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py) and the file-writing logic in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py).

## Frequently Asked Questions

### What happens if I use `--all-chunks` without `--output-dir`?

The CLI will fail because `--all-chunks` requires an output directory to know where to write the files. According to the argument parsing logic in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 34-61), the tool validates that `--output-dir` is provided when `--all-chunks` is enabled.

### Can I control the maximum size of each chunk when using `--all-chunks`?

Yes, combine `--all-chunks` with the `--max-length` parameter. This sets a character limit for each chunk before the splitter creates a new one. For example: `--max-length 20000 --all-chunks --output-dir ./chunks`.

### Does `--all-chunks` work with plain text output instead of HTML?

Yes, add the `--text-only` flag to extract plain text chunks. When enabled, the CLI writes files with `.txt` extensions instead of `.html`, using the `text_render_roi` dictionary from the render system rather than `html_render_roi`.