How to Filter Chunks by Size Using the JSON Output and jq in betterhtmlchunking

Pipe the JSON output from betterhtmlchunking chunk --format json to jq and use select(.html_length <= 10240) or select(.text_length >= 5120) to filter chunks by their pre-calculated character counts.

The betterhtmlchunking library splits HTML documents into semantically coherent chunks for RAG pipelines and content processing. When you need to filter chunks by size using the JSON output and jq, the tool provides built-in length metadata that eliminates the need for manual calculation.

Understanding the JSON Output Structure

When you invoke the chunk command with --format json, the CLI constructs a structured document in betterhtmlchunking/cli.py (lines 97-119). The output contains a "chunks" array where each object includes pre-calculated size metrics.

The schema includes:

• index: Integer identifier starting at 0
• html: Rendered HTML string of the chunk
• text: Plain-text extraction of the chunk
• html_length: Character count of the html field
• text_length: Character count of the text field

Because html_length and text_length are populated during the rendering phase (handled in betterhtmlchunking/render_system.py), you can filter directly on these numeric fields without string manipulation in jq.

Generating JSON Output with the CLI

To enable machine-readable output, pass the --format json flag to the chunk subcommand. The argument parsing occurs in betterhtmlchunking/cli.py (lines 19-24), which validates the format string before execution.

cat page.html | betterhtmlchunking chunk --max-length 32768 --format json

The --max-length parameter controls the maximum allowed size of each region before the algorithm splits it, defaulting to 32768 bytes. This affects the initial chunking strategy, but the final JSON output still contains the exact html_length and text_length values you need for precise filtering.

Filtering Chunks by Size with jq

Once you have the JSON stream, use jq's select function to filter objects based on the length fields. These filters operate entirely on the pre-calculated metadata, making them computationally efficient.

Filtering by HTML Length

To retain only chunks where the HTML representation is smaller than or equal to 10 KB (10240 characters):

cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] | select(.html_length <= 10240)'

Filtering by Text Length

To extract chunks where the plain-text content exceeds 5 KB (5120 characters):

cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] | select(.text_length >= 5120)'

Combining Size Ranges

Use the and operator to specify minimum and maximum boundaries. This example isolates chunks between 4 KB and 8 KB in HTML length:

cat page.html | betterhtmlchunking chunk --format json | \
  jq '.chunks[] |
      select(.html_length >= 4096 and .html_length <= 8192) |
      {index, html}'

Extracting Specific Fields After Filtering

To output only the raw HTML strings of chunks meeting your size criteria, use the -r flag and extract the .html field:

cat page.html | betterhtmlchunking chunk --format json | \
  jq -r '.chunks[] | select(.html_length <= 10240) | .html'

How Chunk Sizes Are Calculated

The html_length and text_length values originate in the rendering pipeline defined in betterhtmlchunking/render_system.py. During chunk generation, the system processes tree regions identified in betterhtmlchunking/tree_regions_system.py (lines 42-62), where the MAX_NODE_REPR_LENGTH constraint from betterhtmlchunking/main.py (lines 21-33) guides the splitting strategy.

However, the final JSON output reflects the actual character counts of the rendered strings, not the byte limits used during segmentation. This distinction allows you to filter post-hoc on precise metrics regardless of the initial --max-length parameter.

Summary

• Use betterhtmlchunking chunk --format json to emit structured output with pre-calculated size metadata.
• The JSON includes html_length and text_length fields for every chunk, populated by the rendering system.
• Pipe the output to jq and use select(.html_length <= 10240) or similar expressions to filter chunks by size.
• Combine conditions with and to define ranges, and project specific fields using jq's object construction syntax.

Frequently Asked Questions

Can I filter chunks by byte size instead of character count?

The html_length and text_length fields represent character counts, not byte sizes. If you need byte-level filtering for multi-byte character sets, pipe the JSON to jq and calculate byte length using (.html | length) for ASCII or encode to UTF-8 bytes first. For most Latin-1 and ASCII HTML documents, character count approximates byte count.

Does the --max-length flag affect the filtering I can do with jq?

The --max-length parameter controls the chunking algorithm's splitting threshold during generation, but it does not restrict your jq filtering. You can filter the resulting JSON for chunks of any size, including those larger than the --max-length value, because the JSON output reflects the actual rendered sizes of the final chunks.

How do I export only the chunk indices that meet my size criteria?

Use jq's object projection to return only the index field. For example: jq '.chunks[] | select(.html_length <= 10240) | {index}'. To get a flat array of indices, wrap the select in square brackets: jq '[.chunks[] | select(.html_length <= 10240) | .index]'.

Can I use these jq filters in a production pipeline with large HTML files?

Yes, because betterhtmlchunking streams the JSON output and jq processes it line-by-line or as a stream. For very large documents, use jq's --stream option or process the chunks array iteratively to avoid loading the entire JSON into memory. The pre-calculated length fields eliminate the need for expensive string operations during filtering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}