performance

Performance Difference Between HTML_LENGTH vs TEXT_LENGTH Chunk Size Comparison Modes

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

The performance difference between HTML_LENGTH and TEXT_LENGTH modes is negligible for most documents because both values are pre-computed during DOM tree construction, with TEXT_LENGTH potentially creating slightly more chunks and loop iterations for very large documents.

The betterhtmlchunking library provides flexible HTML chunking strategies for processing web content. When configuring chunk size limits, you must choose between HTML_LENGTH and TEXT_LENGTH comparison modes, which determine whether the chunker measures raw markup or visible text content.

Understanding the Comparison Modes

The ReprLengthComparisionBy enum in betterhtmlchunking/tree_regions_system.py (lines 52-55) defines these two strategies for measuring node representation length.

HTML_LENGTH Mode

HTML_LENGTH measures the raw HTML markup including tags, attributes, and whitespace. This value is computed in betterhtmlchunking/tree_representation.py (lines 46-50) by taking len(child_html) for each element during the DOMTreeRepresentation.compute_xpaths_data method.

TEXT_LENGTH Mode

TEXT_LENGTH measures only the visible text content, stripping all HTML markup. This is computed by parsel_text.get_bs4_soup_text in betterhtmlchunking/tree_representation.py (lines 41-45), which extracts human-readable text from each node.

How the Modes Affect Chunk Creation

Both modes influence the get_node_repr_length method, which is called during region of interest (ROI) creation in ROIMaker (lines 94-101) and TreeRegionsSystem (lines 33-39) within betterhtmlchunking/tree_regions_system.py.

The algorithm groups child nodes cumulatively, comparing the running total against max_node_repr_length. Since HTML_LENGTH includes markup characters, it typically yields larger per-node values, causing earlier region closures and fewer total chunks. Conversely, TEXT_LENGTH produces smaller values, allowing more nodes per chunk limit and potentially creating more chunks overall.

Performance Impact Analysis

Aspect HTML_LENGTH TEXT_LENGTH
Length values Includes all markup characters; typically larger per node Only visible characters; typically smaller per node
Chunk count Fewer chunks (larger lengths hit limit faster) More chunks (smaller lengths accumulate slower)
Processing time Fewer ROI-building loop iterations More iterations due to additional splits and enqueue/dequeue cycles
Overall impact Negligible for typical documents; sub-millisecond difference Slight overhead for very large documents with many tiny text nodes
Memory usage Identical (both lengths pre-computed and stored) Identical

The dominant performance cost in betterhtmlchunking occurs during initial DOM parsing and tree construction using BeautifulSoup and treelib. Both html_length and text_length are computed once during this phase in betterhtmlchunking/tree_representation.py. The selected comparison mode only affects a single integer lookup per node during ROI grouping, which is O(1) complexity.

Therefore, the performance difference between HTML_LENGTH and TEXT_LENGTH is typically negligible, becoming noticeable only when processing extremely large documents where TEXT_LENGTH generates significantly more chunks, adding marginal loop overhead.

Implementation Examples

You can switch between modes using the CLI or Python API.

Using the CLI with HTML_LENGTH (default):

cat page.html | python -m betterhtmlchunking.cli chunk --max-length 5000 > chunks.html

Using the CLI with TEXT_LENGTH:

cat page.html | python -m betterhtmlchunking.cli chunk --max-length 5000 --text > chunks_text.html

The --text flag toggles the comparison mode in betterhtmlchunking/cli.py (lines 85-88).

Using the Python API:

from betterhtmlchunking.main import DomRepresentation, ReprLengthComparisionBy

html = "<html><body><p>Hello world</p></body></html>"
dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=200,
    website_code=html,
    repr_length_compared_by=ReprLengthComparisionBy.TEXT_LENGTH,  # or HTML_LENGTH

)
dom.start()
print(dom.render_system.html_render_roi)   # rendered chunks

print(dom.render_system.text_render_roi)   # plain-text chunks

Summary

  • Both HTML_LENGTH and TEXT_LENGTH modes pre-compute length values during DOM tree construction in betterhtmlchunking/tree_representation.py, resulting in identical memory overhead.
  • HTML_LENGTH measures raw markup including tags and attributes, typically producing fewer, larger chunks and requiring fewer ROI-building loop iterations.
  • TEXT_LENGTH measures visible text only, often creating more chunks and slightly increasing loop overhead for very large documents.
  • The performance difference is generally negligible (sub-millisecond for typical inputs) because the comparison mode only affects a single O(1) integer lookup per node during region grouping.
  • Choose HTML_LENGTH for markup-aware chunking or TEXT_LENGTH for content-aware splitting based on visible text density.

Frequently Asked Questions

Which mode is faster, HTML_LENGTH or TEXT_LENGTH?

HTML_LENGTH is marginally faster in practice because it typically produces fewer chunks, resulting in fewer iterations of the ROI-building loop. However, the difference is usually sub-millisecond for typical documents because both modes rely on pre-computed length values stored during initial DOM parsing.

When should I use TEXT_LENGTH over HTML_LENGTH?

Use TEXT_LENGTH when you need to limit chunks based on the actual visible content rather than markup overhead. This is ideal for natural language processing tasks, text extraction, or when you want consistent chunk sizes based on readable text density regardless of HTML tag complexity.

Does the choice of comparison mode affect memory usage?

No. Both html_length and text_length are calculated for every node during tree construction in betterhtmlchunking/tree_representation.py and stored in the node data. The comparison mode only determines which pre-computed value is read during ROI grouping, so memory consumption is identical regardless of which mode is active.

How do I switch between modes in the CLI?

Pass the --text flag to use TEXT_LENGTH mode, or omit it to use the default HTML_LENGTH mode. This is handled in betterhtmlchunking/cli.py at lines 85-88, where the flag sets the repr_length_compared_by parameter to ReprLengthComparisionBy.TEXT_LENGTH.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →