How the ROIMaker Class Handles Nodes That Exceed the max_node_repr_length Limit

The ROIMaker class immediately isolates any child node whose representation length meets or exceeds the configured max_node_repr_length, sealing the current region of interest and enqueueing the oversized node for deeper recursive processing.

The ROIMaker class in the carlosplanchon/betterhtmlchunking repository implements a critical safety mechanism to ensure that generated HTML chunks never exceed specified size limits. When processing a DOM node's children, the class must handle nodes that individually exceed the max_node_repr_length threshold without breaking the chunking algorithm's integrity.

The Oversized Node Detection Algorithm

The detection logic resides in betterhtmlchunking/tree_regions_system.py and follows a strict rule-based approach to identify nodes that cannot fit within the current region.

Measuring Node Length with get_node_repr_length

Before processing, ROIMaker determines each child's size using get_node_repr_length (lines 94-101). This method returns either the text or HTML length depending on the repr_length_compared_by setting:

def get_node_repr_length(self, xpath: str) -> int:
    if self.repr_length_compared_by == ReprLengthComparisionBy.TEXT_LENGTH:
        return self.tree_representation.get_node_text_length(xpath=xpath)
    else:
        return self.tree_representation.get_node_html_length(xpath=xpath)

The Iteration Logic in _process_children

The core iteration occurs in _process_children (lines 102-110), which walks through self.children_tags—the XPaths of the node's direct children. For each child, the algorithm evaluates whether the current node can be added to the growing region of interest or if it triggers a boundary condition.

The Critical Threshold Check

The decisive logic for handling oversized nodes appears at lines 113-121. When a child's length meets or exceeds max_node_repr_length, the algorithm immediately triggers isolation:

if node_length >= self.max_node_repr_length:
    if current_region.pos_xpath_list:
        self.regions_of_interest_list.append(current_region)
        current_region = RegionOfInterest()
    self.children_to_enqueue.append(child_xpath)
    continue

This check ensures that no single node exceeding the limit ever enters a region of interest.

Isolating and Enqueueing Oversized Children

When the threshold check triggers, ROIMaker executes a three-step isolation protocol to maintain chunk size integrity.

First, seal the current region. If current_region.pos_xpath_list contains any accumulated children, the region is finalized and appended to self.regions_of_interest_list. This prevents partial regions from remaining open when an oversized sibling interrupts the sequence.

Second, enqueue for recursion. The oversized child's XPath is appended to self.children_to_enqueue. According to the source code in betterhtmlchunking/tree_regions_system.py, the TreeRegionsSystem driver later dequeues these XPaths and instantiates a new ROIMaker for each, effectively descending one level deeper into the DOM tree.

Third, continue iteration. The algorithm proceeds to the next sibling without adding the oversized node to the current region, maintaining the invariant that all regions contain only nodes smaller than the limit.

Practical Implementation Examples

The following examples demonstrate how the oversized node handling operates in practice using the betterhtmlchunking library.

Using TreeRegionsSystem (High-Level API)

For most use cases, interact with the chunking system through TreeRegionsSystem, which internally manages ROIMaker instances and oversized node recursion:

from betterhtmlchunking.tree_representation import DOMTreeRepresentation
from betterhtmlchunking.tree_regions_system import TreeRegionsSystem, ReprLengthComparisionBy

html = "<div><p>Short text</p>" + "<p>" + "x" * 15000 + "</p></div>"
tree_repr = DOMTreeRepresentation(html)

system = TreeRegionsSystem(
    tree_representation=tree_repr,
    max_node_repr_length=10_000,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

chunks = system.sorted_roi_by_pos_xpath
for idx, roi in chunks.items():
    print(f"Chunk {idx}: {len(roi.pos_xpath_list)} nodes, {roi.repr_length} chars")

In this example, the second <p> element exceeds 10,000 characters, so ROIMaker isolates it into children_to_enqueue for separate processing rather than including it in the initial chunk.

Direct ROIMaker Instantiation (Unit Testing)

For testing or custom chunking logic, instantiate ROIMaker directly to observe the oversized node handling:

from betterhtmlchunking.tree_regions_system import ROIMaker, ReprLengthComparisionBy
from betterhtmlchunking.tree_representation import DOMTreeRepresentation

html = "<section><h1>Title</h1><article>" + "Content" * 2000 + "</article></section>"
tree_repr = DOMTreeRepresentation(html)

roi_maker = ROIMaker(
    node_xpath="/html/body/section",
    children_tags=["/html/body/section/h1", "/html/body/section/article"],
    tree_representation=tree_repr,
    max_node_repr_length=1_000,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

print("Regions of Interest:", len(roi_maker.regions_of_interest_list))
print("Oversized nodes to enqueue:", roi_maker.children_to_enqueue)

If the <article> element's HTML length exceeds 5,000 characters, it appears in children_to_enqueue while the <h1> element is safely contained within regions_of_interest_list.

Summary

• Immediate isolation: When ROIMaker encounters a child node with node_length >= max_node_repr_length, it immediately seals the current region and excludes the oversized node.
• Recursive enqueueing: Oversized nodes are stored in children_to_enqueue for the TreeRegionsSystem driver to process in subsequent, deeper recursion levels.
• Invariant preservation: The algorithm guarantees that no region in regions_of_interest_list contains a node exceeding the configured limit, maintaining the integrity of the HTML chunking process.
• Configurable measurement: Node length is determined by get_node_repr_length, which supports both text and HTML length comparisons via the repr_length_compared_by parameter.

Frequently Asked Questions

What happens if a node exactly equals max_node_repr_length?

If a node's length is exactly equal to max_node_repr_length, the condition node_length >= self.max_node_repr_length evaluates to true, triggering the same isolation logic as oversized nodes. The node is enqueued for deeper processing rather than being added to the current region, ensuring strict adherence to the size boundary.

How does ROIMaker prevent oversized nodes from appearing in final regions?

ROIMaker implements a defensive check in the _process_children method (lines 113-121 of betterhtmlchunking/tree_regions_system.py). Before adding any child to the current RegionOfInterest, it verifies the child's length against the threshold. If the check fails, the current region is immediately sealed and appended to regions_of_interest_list, while the oversized child is routed to children_to_enqueue, completely bypassing the region collection.

Can I adjust max_node_repr_length after creating a ROIMaker instance?

No, max_node_repr_length is set during instantiation via the constructor and stored as an instance attribute (self.max_node_repr_length). The class does not provide setter methods or property decorators to modify this value after construction. To use a different limit, you must create a new ROIMaker instance with the desired parameter.

Where does the recursive processing of oversized nodes occur?

The recursive processing is orchestrated by the TreeRegionsSystem class, which acts as the driver for ROIMaker. After ROIMaker populates children_to_enqueue with oversized node XPaths, the TreeRegionsSystem dequeues these paths and instantiates new ROIMaker instances for each one, effectively descending one level deeper into the DOM tree. This driver logic is implemented in the same betterhtmlchunking/tree_regions_system.py file that contains the ROIMaker class.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}