# How to Filter Chunks by Size Using the JSON Output and jq in betterhtmlchunking

> Filter betterhtmlchunking JSON output by size using jq. Learn to select chunks based on html_length or text_length for efficient data processing.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: how-to-guide
- Published: 2026-02-26

---

**Pipe the JSON output from `betterhtmlchunking chunk --format json` to `jq` and use `select(.html_length <= 10240)` or `select(.text_length >= 5120)` to filter chunks by their pre-calculated character counts.**

The `betterhtmlchunking` library splits HTML documents into semantically coherent chunks for RAG pipelines and content processing. When you need to filter chunks by size using the JSON output and jq, the tool provides built-in length metadata that eliminates the need for manual calculation.

## Understanding the JSON Output Structure

When you invoke the `chunk` command with `--format json`, the CLI constructs a structured document in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 97-119). The output contains a `"chunks"` array where each object includes pre-calculated size metrics.

The schema includes:

- **index**: Integer identifier starting at 0
- **html**: Rendered HTML string of the chunk
- **text**: Plain-text extraction of the chunk
- **html_length**: Character count of the `html` field
- **text_length**: Character count of the `text` field

Because `html_length` and `text_length` are populated during the rendering phase (handled in [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py)), you can filter directly on these numeric fields without string manipulation in jq.

## Generating JSON Output with the CLI

To enable machine-readable output, pass the `--format json` flag to the `chunk` subcommand. The argument parsing occurs in [`betterhtmlchunking/cli.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/cli.py) (lines 19-24), which validates the format string before execution.

```bash
cat page.html | betterhtmlchunking chunk --max-length 32768 --format json

```

The `--max-length` parameter controls the maximum allowed size of each region before the algorithm splits it, defaulting to 32768 bytes. This affects the initial chunking strategy, but the final JSON output still contains the exact `html_length` and `text_length` values you need for precise filtering.

## Filtering Chunks by Size with jq

Once you have the JSON stream, use jq's `select` function to filter objects based on the length fields. These filters operate entirely on the pre-calculated metadata, making them computationally efficient.

### Filtering by HTML Length

To retain only chunks where the HTML representation is smaller than or equal to 10 KB (10240 characters):

```bash
cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] | select(.html_length <= 10240)'

```

### Filtering by Text Length

To extract chunks where the plain-text content exceeds 5 KB (5120 characters):

```bash
cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] | select(.text_length >= 5120)'

```

### Combining Size Ranges

Use the `and` operator to specify minimum and maximum boundaries. This example isolates chunks between 4 KB and 8 KB in HTML length:

```bash
cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] |
      select(.html_length >= 4096 and .html_length <= 8192) |
      {index, html}'

```

### Extracting Specific Fields After Filtering

To output only the raw HTML strings of chunks meeting your size criteria, use the `-r` flag and extract the `.html` field:

```bash
cat page.html | betterhtmlchunking chunk --format json | \
  jq -r '.chunks[] | select(.html_length <= 10240) | .html'

```

## How Chunk Sizes Are Calculated

The `html_length` and `text_length` values originate in the rendering pipeline defined in [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py). During chunk generation, the system processes tree regions identified in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py) (lines 42-62), where the `MAX_NODE_REPR_LENGTH` constraint from [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py) (lines 21-33) guides the splitting strategy.

However, the final JSON output reflects the actual character counts of the rendered strings, not the byte limits used during segmentation. This distinction allows you to filter post-hoc on precise metrics regardless of the initial `--max-length` parameter.

## Summary

- Use `betterhtmlchunking chunk --format json` to emit structured output with pre-calculated size metadata.
- The JSON includes `html_length` and `text_length` fields for every chunk, populated by the rendering system.
- Pipe the output to `jq` and use `select(.html_length <= 10240)` or similar expressions to filter chunks by size.
- Combine conditions with `and` to define ranges, and project specific fields using jq's object construction syntax.

## Frequently Asked Questions

### Can I filter chunks by byte size instead of character count?

The `html_length` and `text_length` fields represent character counts, not byte sizes. If you need byte-level filtering for multi-byte character sets, pipe the JSON to jq and calculate byte length using `(.html | length)` for ASCII or encode to UTF-8 bytes first. For most Latin-1 and ASCII HTML documents, character count approximates byte count.

### Does the `--max-length` flag affect the filtering I can do with jq?

The `--max-length` parameter controls the chunking algorithm's splitting threshold during generation, but it does not restrict your jq filtering. You can filter the resulting JSON for chunks of any size, including those larger than the `--max-length` value, because the JSON output reflects the actual rendered sizes of the final chunks.

### How do I export only the chunk indices that meet my size criteria?

Use jq's object projection to return only the `index` field. For example: `jq '.chunks[] | select(.html_length <= 10240) | {index}'`. To get a flat array of indices, wrap the select in square brackets: `jq '[.chunks[] | select(.html_length <= 10240) | .index]'`.

### Can I use these jq filters in a production pipeline with large HTML files?

Yes, because `betterhtmlchunking` streams the JSON output and jq processes it line-by-line or as a stream. For very large documents, use jq's `--stream` option or process the chunks array iteratively to avoid loading the entire JSON into memory. The pre-calculated length fields eliminate the need for expensive string operations during filtering.