# How to Use the JSON Output Format in betterhtmlchunking for Programmatic Processing with jq

> Learn to use JSON output format in betterhtmlchunking for programmatic processing. Pipe clean stdout to jq for easy filtering and data extraction.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Activate JSON mode by passing `--format json` (or `-f json`) to the CLI, then pipe the clean stdout stream to `jq` for filtering and extraction.**

The `betterhtmlchunking` library splits HTML documents into manageable chunks for LLM processing. When you need to automate workflows or extract specific metadata, the **JSON output format** provides a structured, machine-readable alternative to plain text output.

## Enabling JSON Output Mode

To switch from the default human-readable output to machine-readable JSON, append the `--format json` flag to any CLI invocation. The short form `-f json` is also accepted.

When this flag is present, the CLI bypasses standard output formatting and constructs a Python dictionary named `output`. This dictionary is serialized with `json.dumps(..., indent=2, ensure_ascii=False)` and printed to **stdout**. All diagnostic logs are emitted to **stderr**, ensuring the JSON stream remains pure and safe to pipe directly into tools like `jq` without filtering noise.

## Understanding the JSON Schema Structure

The JSON document returned by `betterhtmlchunking` contains top-level metadata about the chunking operation and an array of individual chunk objects.

### Top-Level Metadata Fields

According to the implementation in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 112-114), the root object includes:

- **`total_chunks`**: Integer count of chunks produced for the input document.
- **`max_length`**: The `--max-length` value supplied by the user, indicating the size constraint applied during chunking.
- **`compared_by`**: String value of `"html"` if chunk size was measured on raw HTML, or `"text"` if measured on rendered text (triggered by the `--text` flag).

### The Chunks Array

The `chunks` field contains an array where each element describes a single chunk. As implemented in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 104-108), each chunk object provides:

- **`index`**: Integer position in the DOM traversal order.
- **`html`**: The raw HTML markup for the chunk.
- **`text`**: The plain-text rendering of the same chunk with tags stripped.
- **`html_length`**: Character count of the `html` string.
- **`text_length`**: Character count of the `text` string.

## Practical jq Examples for betterhtmlchunking JSON

The following examples demonstrate how to process `betterhtmlchunking` JSON output using `jq` in Unix pipelines.

### Extracting Total Chunk Count

To retrieve just the number of chunks generated:

```bash
cat my_page.html | \
  betterhtmlchunking --max-length 5000 --format json | \
  jq '.total_chunks'

```

This outputs a single integer (e.g., `12`) suitable for scripting conditionals.

### Filtering Chunks by Index

To extract the HTML content of a specific chunk (e.g., index 2):

```bash
cat my_page.html | \
  betterhtmlchunking --max-length 4000 --format json | \
  jq -r '.chunks[] | select(.index == 2) | .html'

```

The `-r` flag returns raw text, removing JSON string quotes and escaping, delivering clean HTML markup.

### Exporting HTML to Individual Files

To save each chunk to its own file:

```bash
cat my_page.html | \
  betterhtmlchunking --max-length 2500 --format json | \
  jq -r '.chunks[] | "chunk_\(.index).html \(.html)"' | \
  while read -r filename content; do
    printf '%s' "$content" > "$filename"
  done

```

This creates [`chunk_0.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_0.html), [`chunk_1.html`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/chunk_1.html), etc., each containing the exact HTML chunk.

### Processing Text Content Streams

To stream each chunk's plain text with custom prefixes:

```bash
cat my_page.html | \
  betterhtmlchunking --max-length 2000 --format json | \
  jq -c '.chunks[]' | \
  while read -r chunk; do
    echo "$chunk" | jq -r '.text' | sed -e 's/^/>>> /'
  done

```

This outputs each chunk's text content prefixed with `>>> `, useful for debugging or feeding into text-processing pipelines.

## Key Implementation Details

The JSON output functionality is implemented in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py). The code constructs the `output` dictionary during the chunking process and serializes it using Python's standard library:

```python
json.dumps(output, indent=2, ensure_ascii=False)

```

This ensures human-readable indentation while preserving Unicode characters without escaping. The separation of concerns—JSON to `stdout` and logs to `stderr`—is verified in [`tests/test_cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/tests/test_cli.py), which confirms that the JSON schema contains all required fields and that diagnostic output does not contaminate the data stream.

## Summary

- Activate **JSON output format** with `--format json` or `-f json` when running the `betterhtmlchunking` CLI.
- The JSON schema includes **metadata** (`total_chunks`, `max_length`, `compared_by`) and a **chunks array** with `html`, `text`, and length metrics for each chunk.
- Output is sent to **stdout** while logs go to **stderr**, enabling safe piping to `jq` and other JSON processors.
- Use `jq` selectors like `.chunks[]`, `select(.index == N)`, and `-r` for raw string output to extract specific chunks or metadata.

## Frequently Asked Questions

### How do I activate JSON output format in betterhtmlchunking?

Pass the `--format json` flag (or `-f json` shorthand) to the CLI command. This instructs the tool to emit a structured JSON document instead of the default human-readable text format.

### What fields are included in the JSON output?

The root object contains `total_chunks` (integer), `max_length` (integer), and `compared_by` (string). The `chunks` array contains objects with `index`, `html`, `text`, `html_length`, and `text_length` fields, as implemented in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py).

### Can I pipe betterhtmlchunking JSON output directly to jq?

Yes. Because the tool writes the JSON payload to **stdout** and diagnostic logs to **stderr**, you can safely pipe the output directly into `jq` without filtering. For example: `betterhtmlchunking --format json < page.html | jq '.total_chunks'`.

### How does betterhtmlchunking handle logging when outputting JSON?

All log messages are emitted to **stderr**, while the JSON data stream is written to **stdout**. This separation ensures that JSON parsers receive a clean, valid document even when verbose logging is enabled, as verified in [`tests/test_cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/tests/test_cli.py).