How to Process HTML from Stdin Using the BetterHTMLChunking CLI and Pipe Output to Other Tools

The BetterHTMLChunking CLI reads raw HTML from standard input via sys.stdin.read() in cli.py, processes it through the DOM representation and chunking pipeline, and writes results to stdout, enabling seamless integration with Unix pipes and filters like jq, grep, and sed.

The BetterHTMLChunking package provides a lightweight command-line interface for splitting HTML documents into manageable chunks. When you need to process HTML from stdin using the BetterHTMLChunking CLI, the tool's design around standard I/O channels makes it ideal for shell pipelines and automated workflows. According to the carlosplanchon/betterhtmlchunking source code, the CLI never writes to files unless explicitly requested with --all-chunks, keeping the standard pipeline pure.

How the CLI Reads from Standard Input

At the entry point in betterhtmlchunking/cli.py, the command invokes sys.stdin.read() at line 83 to collect the complete HTML document piped in from upstream tools.

# Conceptual representation of cli.py line 83

html_content = sys.stdin.read()

This raw HTML string is then handed to the DomRepresentation class defined in betterhtmlchunking/main.py. The DomRepresentation.__attrs_post_init__ method (lines 70-75) constructs a DOM tree, removes unwanted tags via the utility functions, and prepares the data for the chunking algorithm.

The Chunking Pipeline Architecture

Once the CLI receives input from stdin, it executes a three-step pipeline before emitting to stdout:

1. DOM Construction and Cleaning: The DomRepresentation class parses the raw HTML and strips non-essential tags as implemented in betterhtmlchunking/utils.py.

2. Region of Interest Calculation: The TreeRegionsSystem class (from betterhtmlchunking/tree_regions_system.py) analyzes the DOM tree and identifies split points based on the --max-length parameter you provide.

3. Rendering: The RenderSystem class (from betterhtmlchunking/render_system.py) executes automatically via its __attrs_post_init__ method (lines 30-31), transforming each ROI into a self-contained HTML snippet and an equivalent plain-text version.

Output Modes for Pipeline Integration

The CLI supports multiple output formats controlled by flags in cli.py (lines 72-73, 98-119), determining what gets written to stdout:

• Single Chunk Output (default): Prints the HTML for chunk index 0, or the index specified with --chunk-index.

• JSON Envelope (--format json): Wraps all chunks in a structured JSON object suitable for parsing with jq or other JSON processors.

• Text-Only Mode (--text-only): Outputs the plain-text representation instead of HTML, ideal for grep or text processing.

• Chunk Statistics (--list-chunks): Emits a summary like "Total chunks: 7" to help you determine processing scope.

• Batch Export (--all-chunks): Writes individual files to --output-dir, bypassing stdout for file-based workflows.

Practical Shell Pipeline Examples

Because the BetterHTMLChunking CLI uses stdin and stdout exclusively, it integrates seamlessly with standard Unix tools.

Extract and Save the First Chunk

cat page.html | betterhtmlchunking > first_chunk.html

This reads page.html via stdin, processes it through the pipeline in main.py, and redirects the default chunk (index 0) to a file.

Parse JSON Output with jq

cat page.html \
  | betterhtmlchunking --format json \
  | jq '.chunks[] | {index, html_length, text_length}'

The --format json flag (handled in cli.py lines 98-119) structures the output, allowing jq to extract specific fields from the rendered chunks.

Text Processing with grep and wc

cat page.html \
  | betterhtmlchunking --chunk-index 3 --text-only \
  | grep -i "warning" \
  | wc -l

This extracts the text-only version of chunk 3 using the text rendering system from render_system.py, then pipes it through grep to filter for "warning" and wc to count matches.

Stream Processing with sed

cat page.html \
  | betterhtmlchunking --text-only \
  | sed 's/{{NAME}}/Alice/g'

After the CLI outputs plain text via stdout, sed performs in-stream substitution before the data reaches its final destination.

Python Post-Processing

cat page.html \
  | betterhtmlchunking --format json \
  | python -c "import sys, json; data=json.load(sys.stdin); print([c['index'] for c in data['chunks'] if 'login' in c['text'].lower()])"

The JSON output mode enables complex filtering using Python one-liners or scripts, accessing both the HTML and text representations generated by the RenderSystem.

Summary

• The BetterHTMLChunking CLI reads HTML from stdin using sys.stdin.read() in cli.py at line 83, making it compatible with any tool that outputs HTML.

• The processing pipeline involves DomRepresentation (DOM construction), TreeRegionsSystem (ROI calculation), and RenderSystem (HTML/text generation) as implemented across main.py, tree_regions_system.py, and render_system.py.

• Output defaults to stdout unless you use --all-chunks, enabling seamless pipes to jq, grep, sed, awk, and other Unix utilities.

• Use --format json for structured data interchange, --text-only for plain text processing, and --chunk-index to select specific chunks from the stream.

Frequently Asked Questions

Can I process multiple HTML files in a loop using the CLI?

Yes. Because the BetterHTMLChunking CLI accepts stdin, you can loop over files and pipe each one individually. Use a bash loop like for f in *.html; do cat "$f" | betterhtmlchunking --format json >> results.jsonl; done to process multiple documents and append the JSON output to a single line-delimited file.

Does the CLI support streaming partial HTML or only complete documents?

The current implementation in cli.py uses sys.stdin.read() which buffers the entire input before processing. The DomRepresentation class in main.py requires a complete HTML document to build the DOM tree in __attrs_post_init__. Therefore, you must pipe complete documents rather than streaming fragments.

How do I extract all chunks at once without writing to individual files?

Use the --format json flag to emit all chunks as a JSON array to stdout, then pipe to your processing tool. For example: cat page.html | betterhtmlchunking --format json | jq '.chunks[].html'. This avoids the file-creation behavior of --all-chunks while still giving you access to every chunk generated by the TreeRegionsSystem.

What happens if the HTML contains no splittable regions?

If the TreeRegionsSystem in tree_regions_system.py cannot identify multiple regions of interest based on your --max-length threshold, the CLI will treat the entire document as a single chunk. The RenderSystem will still generate both HTML and text representations, and the output will contain one chunk object (or the single chunk content, depending on your flags).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}