# How to Process HTML from Stdin Using the BetterHTMLChunking CLI and Pipe Output to Other Tools

> Process HTML from stdin using the BetterHTMLChunking CLI. Pipe output to tools like jq and grep for powerful text manipulation. Integrate seamlessly with Unix pipes.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: how-to-guide
- Published: 2026-02-26

---

**The BetterHTMLChunking CLI reads raw HTML from standard input via `sys.stdin.read()` in [`cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/cli.py), processes it through the DOM representation and chunking pipeline, and writes results to stdout, enabling seamless integration with Unix pipes and filters like `jq`, `grep`, and `sed`.**

The BetterHTMLChunking package provides a lightweight command-line interface for splitting HTML documents into manageable chunks. When you need to process HTML from stdin using the BetterHTMLChunking CLI, the tool's design around standard I/O channels makes it ideal for shell pipelines and automated workflows. According to the carlosplanchon/betterhtmlchunking source code, the CLI never writes to files unless explicitly requested with `--all-chunks`, keeping the standard pipeline pure.

## How the CLI Reads from Standard Input

At the entry point in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py), the command invokes `sys.stdin.read()` at line 83 to collect the complete HTML document piped in from upstream tools.

```python

# Conceptual representation of cli.py line 83

html_content = sys.stdin.read()

```

This raw HTML string is then handed to the **DomRepresentation** class defined in [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py). The `DomRepresentation.__attrs_post_init__` method (lines 70-75) constructs a DOM tree, removes unwanted tags via the utility functions, and prepares the data for the chunking algorithm.

## The Chunking Pipeline Architecture

Once the CLI receives input from stdin, it executes a three-step pipeline before emitting to stdout:

1. **DOM Construction and Cleaning**: The `DomRepresentation` class parses the raw HTML and strips non-essential tags as implemented in [`betterhtmlchunking/utils.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/utils.py).

2. **Region of Interest Calculation**: The `TreeRegionsSystem` class (from [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py)) analyzes the DOM tree and identifies split points based on the `--max-length` parameter you provide.

3. **Rendering**: The `RenderSystem` class (from [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py)) executes automatically via its `__attrs_post_init__` method (lines 30-31), transforming each ROI into a self-contained HTML snippet and an equivalent plain-text version.

## Output Modes for Pipeline Integration

The CLI supports multiple output formats controlled by flags in [`cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/cli.py) (lines 72-73, 98-119), determining what gets written to stdout:

- **Single Chunk Output** (default): Prints the HTML for chunk index 0, or the index specified with `--chunk-index`.

- **JSON Envelope** (`--format json`): Wraps all chunks in a structured JSON object suitable for parsing with `jq` or other JSON processors.

- **Text-Only Mode** (`--text-only`): Outputs the plain-text representation instead of HTML, ideal for `grep` or text processing.

- **Chunk Statistics** (`--list-chunks`): Emits a summary like "Total chunks: 7" to help you determine processing scope.

- **Batch Export** (`--all-chunks`): Writes individual files to `--output-dir`, bypassing stdout for file-based workflows.

## Practical Shell Pipeline Examples

Because the BetterHTMLChunking CLI uses stdin and stdout exclusively, it integrates seamlessly with standard Unix tools.

### Extract and Save the First Chunk

```bash
cat page.html | betterhtmlchunking > first_chunk.html

```

This reads [`page.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/page.html) via stdin, processes it through the pipeline in [`main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/main.py), and redirects the default chunk (index 0) to a file.

### Parse JSON Output with jq

```bash
cat page.html \
  | betterhtmlchunking --format json \
  | jq '.chunks[] | {index, html_length, text_length}'

```

The `--format json` flag (handled in [`cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/cli.py) lines 98-119) structures the output, allowing `jq` to extract specific fields from the rendered chunks.

### Text Processing with grep and wc

```bash
cat page.html \
  | betterhtmlchunking --chunk-index 3 --text-only \
  | grep -i "warning" \
  | wc -l

```

This extracts the text-only version of chunk 3 using the text rendering system from [`render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/render_system.py), then pipes it through `grep` to filter for "warning" and `wc` to count matches.

### Stream Processing with sed

```bash
cat page.html \
  | betterhtmlchunking --text-only \
  | sed 's/{{NAME}}/Alice/g'

```

After the CLI outputs plain text via stdout, `sed` performs in-stream substitution before the data reaches its final destination.

### Python Post-Processing

```bash
cat page.html \
  | betterhtmlchunking --format json \
  | python -c "import sys, json; data=json.load(sys.stdin); print([c['index'] for c in data['chunks'] if 'login' in c['text'].lower()])"

```

The JSON output mode enables complex filtering using Python one-liners or scripts, accessing both the HTML and text representations generated by the `RenderSystem`.

## Summary

- The BetterHTMLChunking CLI reads HTML from **stdin** using `sys.stdin.read()` in [`cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/cli.py) at line 83, making it compatible with any tool that outputs HTML.

- The processing pipeline involves **DomRepresentation** (DOM construction), **TreeRegionsSystem** (ROI calculation), and **RenderSystem** (HTML/text generation) as implemented across [`main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/main.py), [`tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/tree_regions_system.py), and [`render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/render_system.py).

- Output defaults to **stdout** unless you use `--all-chunks`, enabling seamless pipes to `jq`, `grep`, `sed`, `awk`, and other Unix utilities.

- Use **`--format json`** for structured data interchange, **`--text-only`** for plain text processing, and **`--chunk-index`** to select specific chunks from the stream.

## Frequently Asked Questions

### Can I process multiple HTML files in a loop using the CLI?

Yes. Because the BetterHTMLChunking CLI accepts stdin, you can loop over files and pipe each one individually. Use a bash loop like `for f in *.html; do cat "$f" | betterhtmlchunking --format json >> results.jsonl; done` to process multiple documents and append the JSON output to a single line-delimited file.

### Does the CLI support streaming partial HTML or only complete documents?

The current implementation in [`cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/cli.py) uses `sys.stdin.read()` which buffers the entire input before processing. The `DomRepresentation` class in [`main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/main.py) requires a complete HTML document to build the DOM tree in `__attrs_post_init__`. Therefore, you must pipe complete documents rather than streaming fragments.

### How do I extract all chunks at once without writing to individual files?

Use the `--format json` flag to emit all chunks as a JSON array to stdout, then pipe to your processing tool. For example: `cat page.html | betterhtmlchunking --format json | jq '.chunks[].html'`. This avoids the file-creation behavior of `--all-chunks` while still giving you access to every chunk generated by the `TreeRegionsSystem`.

### What happens if the HTML contains no splittable regions?

If the `TreeRegionsSystem` in [`tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/tree_regions_system.py) cannot identify multiple regions of interest based on your `--max-length` threshold, the CLI will treat the entire document as a single chunk. The `RenderSystem` will still generate both HTML and text representations, and the output will contain one chunk object (or the single chunk content, depending on your flags).