internals

What Happens When max_node_repr_length Is Larger Than the Document in BetterHTMLChunking

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

When the max_node_repr_length threshold exceeds the total representation length of the HTML document, the Tree Regions System automatically falls back to treating the entire document as a single Region of Interest (ROI) instead of returning an empty list.

The betterhtmlchunking library by carlosplanchon partitions HTML documents into manageable chunks based on representation length limits. When you configure a max_node_repr_length value larger than the document itself, the algorithm cannot create smaller sub-regions because every node—including the root—fits comfortably within your specified threshold. In this edge case, the system gracefully degrades to a single-ROI strategy to ensure downstream processing always has valid content to work with.

How the Tree Regions System Creates ROIs

The Tree Regions System processes the DOM tree using a breadth-first search (BFS) traversal to group child nodes into regions of interest (ROIs). Each ROI represents a cluster of nodes whose combined representation length does not exceed the user-supplied max_node_repr_length threshold.

During normal operation, the system evaluates node representation lengths—whether measured by HTML character count, text content, or token count—and partitions the document accordingly. However, when your threshold is set higher than the entire document's representation length, no valid subdivision points exist. The system detects this condition after sorting the collected ROI objects and finding an empty list despite the DOM containing valid nodes.

Fallback Behavior When No ROI Is Found Due to max_node_repr_length Settings

When max_node_repr_length is configured larger than the document size, no ROIs are generated during the initial traversal. In this scenario, the library executes a specific fallback block inside betterhtmlchunking/tree_regions_system.py, where the TreeRegionsSystem.start method detects the empty sorted list while the DOM contains at least one node, then constructs a single RegionOfInterest spanning the root node.

The fallback logic uses the first XPath in pos_xpaths_list (which corresponds to the document root) and creates a synthetic ROI covering the entire document:


# betterhtmlchunking/tree_regions_system.py

# … after processing all nodes …

if sorted_regions == [] and len(self.tree_representation.pos_xpaths_list) > 0:
    logger.info(
        "No ROIs found with current settings, using entire document as single ROI"
    )
    node_xpath = self.tree_representation.pos_xpaths_list[0]

    node = self.tree_representation.tree.get_node(node_xpath)
    node_repr_length = self.get_node_repr_length(node)

    roi = RegionOfInterest()
    roi.pos_xpath_list = [node_xpath]
    roi.repr_length = node_repr_length
    roi.node_is_roi = True

    sorted_regions = [roi]

Source: TreeRegionsSystem.start implementation

This implementation guarantees that downstream rendering and chunking operations always receive at least one valid region, preventing pipeline failures due to empty ROI lists.

Code Examples: Triggering vs. Avoiding the Fallback

Example 1: Triggering the Single-ROI Fallback

Setting MAX_NODE_REPR_LENGTH to a value far exceeding your document size forces the fallback mechanism. The system logs an informational message and returns the root node as the sole ROI:

from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.tree_regions_system import ReprLengthComparisionBy

# Small HTML document

html = "<html><body><p>Hello world</p></body></html>"

# Threshold far larger than document size

doc = DomRepresentation(
    MAX_NODE_REPR_LENGTH=10_000,      # exceeds total document length

    website_code=html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

doc.start(maximal_verbose=True)

# Inspect the resulting ROI

roi = list(doc.tree_regions_system.sorted_roi_by_pos_xpath.values())[0]
print("ROI XPaths :", roi.pos_xpath_list)
print("ROI length :", roi.repr_length)
print("Is whole node ROI :", roi.node_is_roi)

Output:


ROI XPaths : ['/html']
ROI length : 54          # total HTML length of the document

Is whole node ROI : True

The log output confirms the fallback activation:


INFO  No ROIs found with current settings, using entire document as single ROI

Example 2: Normal Multi-ROI Behavior

With a sensible max_node_repr_length value smaller than the document, the system creates multiple ROIs as expected:

doc = DomRepresentation(
    MAX_NODE_REPR_LENGTH=30,   # smaller than total document size

    website_code=html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

doc.start()
print(len(doc.tree_regions_system.sorted_roi_by_pos_xpath))

# Output: 2 (or more, depending on HTML structure)

Here the algorithm successfully subdivides the DOM because the limit forces the creation of smaller, distinct chunks.

Summary

  • Automatic fallback: When max_node_repr_length exceeds the document's total representation length, the system cannot create smaller ROIs and automatically treats the entire document as a single region.
  • Root node coverage: The fallback ROI always uses the first XPath in pos_xpaths_list, which corresponds to the document root (/html), ensuring complete content coverage.
  • Zero pipeline failures: The TreeRegionsSystem.start method in betterhtmlchunking/tree_regions_system.py guarantees at least one ROI exists before proceeding to rendering and chunking stages.
  • Observable behavior: The fallback triggers an INFO-level log message ("No ROIs found with current settings...") and sets node_is_roi=True on the resulting region object.

Frequently Asked Questions

What is the max_node_repr_length parameter in BetterHTMLChunking?

The max_node_repr_length parameter defines the maximum allowed representation length for any single Region of Interest (ROI) during the HTML chunking process. According to the source code in betterhtmlchunking/main.py, this threshold controls how the DomRepresentation class partitions the DOM tree, with the comparison method determined by repr_length_compared_by (HTML length, text length, or token count).

Why does the library return the entire document as a single ROI instead of an empty list?

The library implements a defensive programming pattern in TreeRegionsSystem.start to prevent downstream pipeline failures. When sorted_regions is empty but the DOM contains nodes, the code constructs a synthetic RegionOfInterest covering the root node. This ensures that rendering and chunking operations always have valid content to process, avoiding null pointer exceptions or empty iteration errors in consumer code.

How can I detect when the fallback mechanism has been triggered?

You can detect fallback activation through two indicators: first, the system logs the message "No ROIs found with current settings, using entire document as single ROI" at INFO level when maximal_verbose=True. Second, inspect the resulting ROI object—if node_is_roi is True and the pos_xpath_list contains only the root XPath (/html), the fallback has been applied.

What happens if I set max_node_repr_length to an extremely large value like 100,000?

Setting max_node_repr_length to 100,000 on a typical HTML document will almost certainly trigger the fallback mechanism described in betterhtmlchunking/tree_regions_system.py. Since most web pages have representation lengths far below this threshold, the algorithm cannot identify valid subdivision points and returns the entire document as a single ROI with the root node's XPath, effectively disabling granular chunking for that document.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →