How the RegionOfInterest (ROI) System Groups DOM Nodes into Semantically Coherent Chunks

The RegionOfInterest system groups DOM nodes into semantically coherent chunks by traversing the document tree breadth-first, packing siblings into size-constrained regions, and promoting parent elements to single chunks when their children fit entirely within the threshold.

The betterhtmlchunking library solves the problem of splitting complex HTML documents into manageable, contextually meaningful segments. At its heart, the RegionOfInterest (ROI) system employs a three-stage pipeline that walks the DOM tree, evaluates node representations, and assembles logical chunks without fragmenting semantic boundaries. This mechanism ensures that related content—such as complete paragraphs or list items—remains grouped even when enforcing strict size limits for downstream processing like summarization or translation.

The Three-Stage ROI Creation Pipeline

The ROI creation process operates through tightly coupled stages defined in betterhtmlchunking/tree_regions_system.py, utilizing a breadth-first strategy to maintain document order and sibling relationships.

Stage 1: Breadth-First Tree Traversal via TreeRegionsSystem.start

The traversal engine initializes with a FIFO queue (queue.Queue) that manages XPaths requiring examination. According to the source code in betterhtmlchunking/tree_regions_system.py, the algorithm begins by selecting the root node (/html if present, otherwise the first available XPath) and placing it on the queue.

While the queue contains nodes, the system:

1. Pops the next XPath and retrieves its direct children via get_children_tag_list
2. Delegates child processing to the ROIMaker class
3. Collects all RegionOfInterest objects returned by the maker
4. Re-queues any children too large to fit into current chunks for deeper processing

After the loop completes, the raw ROI list undergoes sorting by document order using order_regions_of_interest_by_pos_xpath. If no generated ROI satisfies the size limit, the system collapses the entire document into a single RegionOfInterest.

Stage 2: Size-Constrained Chunk Packing in ROIMaker._process_children

For each set of siblings retrieved during traversal, ROIMaker._process_children evaluates whether they can join the current chunk without exceeding max_node_repr_length. The method queries node representation lengths via TreeRegionsSystem.get_node_repr_length, which returns either text length or HTML length based on the repr_length_compared_by configuration (ReprLengthComparisionBy.TEXT_LENGTH or ReprLengthComparisionBy.HTML_LENGTH).

The packing logic enforces these rules:

• If a single child exceeds the limit, it bypasses the current chunk and enters children_to_enqueue for deeper decomposition
• When adding a child would cross the threshold, the current ROI closes and a new one initializes
• After processing all children, any leftover region merges with the previous chunk (if one exists) or stands as the sole region

This greedy packing strategy ensures that sequential siblings appearing together in markup—such as consecutive <li> elements—remain grouped when their combined size permits.

Stage 3: Semantic Node Promotion via _check_node_as_roi

The final stage determines whether a parent node should collapse into a single RegionOfInterest. In ROIMaker._check_node_as_roi, the system evaluates if a node has no children or if all children occupy a single region whose total length equals the node's full representation length.

When these conditions meet, the algorithm sets node_is_roi = True and creates a RegionOfInterest containing only the parent node's XPath. This promotion preserves higher-level semantics by keeping intact structures—such as a <section> containing only small inline elements—as unified chunks rather than fragmenting them across multiple regions.

How Semantic Coherence Is Achieved

The ROI system maintains semantic integrity through four specific design decisions:

• Size-based packing with configurable limits: By enforcing max_node_repr_length, the system prevents oversized chunks that would break rendering pipelines while keeping related content together when possible.

• Breadth-first sibling processing: Examining nodes level-by-level ensures that semantically related siblings (e.g., list items, table cells) are considered for grouping before the algorithm descends into their children.

• Parent-node promotion: When a subtree fits entirely within constraints, the _check_node_as_roi method collapses it into a single chunk, preserving the document's hierarchical meaning rather than splitting a cohesive element arbitrarily.

• Flexible length calculation: The repr_length_compared_by parameter allows callers to choose between HTML_LENGTH (character count including tags) or TEXT_LENGTH (visible content only), adapting chunking behavior for use cases ranging from safe HTML rendering to text summarization.

Practical Implementation Examples

The following examples demonstrate how to configure and execute the ROI system using the high-level DomRepresentation façade and the low-level TreeRegionsSystem directly.

from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.tree_regions_system import ReprLengthComparisionBy

# Example HTML input

html = """
<article>
  <h1>Title</h1>
  <p>This is a short paragraph.</p>
  <p>Another paragraph with <strong>bold</strong> text.</p>
  <img src="pic.jpg"/>
</article>
"""

# Build the full pipeline – each chunk will be ≤ 150 characters of HTML

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=150,
    website_code=html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

# Run the pipeline

dom.start(verbose=True, maximal_verbose=False)

# Access the generated ROIs (chunks)

for idx, roi in dom.tree_regions_system.sorted_roi_by_pos_xpath.items():
    print(f"Chunk {idx}:")
    print("  XPaths:", roi.pos_xpath_list)
    print("  HTML length:", roi.repr_length)

# Direct low‑level usage without the full DomRepresentation wrapper

from betterhtmlchunking.tree_representation import DOMTreeRepresentation
from betterhtmlchunking.tree_regions_system import TreeRegionsSystem, ReprLengthComparisionBy

tree_repr = DOMTreeRepresentation(website_code=html)
tree_repr.recompute_representation()

roi_system = TreeRegionsSystem(
    tree_representation=tree_repr,
    max_node_repr_length=200,                     # limit per chunk

    repr_length_compared_by=ReprLengthComparisionBy.TEXT_LENGTH,
)

# After construction the ROI list is ready

for i, roi in roi_system.sorted_roi_by_pos_xpath.items():
    print(f"ROI {i} – {len(roi.pos_xpath_list)} nodes, text length {roi.repr_length}")

Core Source Files and Architecture

Understanding the ROI system requires familiarity with these key components in the betterhtmlchunking repository:

• betterhtmlchunking/tree_regions_system.py: Contains the core ROI creation logic, including TreeRegionsSystem, ROIMaker, and the document ordering utilities.

• betterhtmlchunking/tree_representation.py: Builds a treelib tree from BeautifulSoup and provides node metadata such as text_length and html_length used during size calculations.

• betterhtmlchunking/main.py: Implements the high-level DomRepresentation façade that orchestrates the tree representation, ROI system, and rendering pipeline.

• betterhtmlchunking/utils.py: Provides preprocessing utilities that remove unwanted tags (such as <script> and <style>) before ROI processing begins.

• betterhtmlchunking/render_system.py: Consumes the final ROI list to produce HTML or text chunks suitable for downstream applications like translation services or content summarization.

Summary

• The RegionOfInterest system in betterhtmlchunking uses a breadth-first traversal (TreeRegionsSystem.start) to process DOM nodes in document order while maintaining a queue for oversized elements requiring deeper inspection.

• Chunk creation occurs in ROIMaker._process_children, which greedily packs sibling nodes into regions until reaching max_node_repr_length, using either HTML or text length calculations based on the repr_length_compared_by setting.

• Semantic promotion via _check_node_as_roi collapses entire subtrees into single chunks when their children fit within constraints, preventing fragmentation of cohesive elements like paragraphs or sections.

• The system guarantees semantic coherence by processing siblings together, respecting size limits, and preserving parent-node integrity when possible.

Frequently Asked Questions

What triggers a node to be treated as its own RegionOfInterest?

A node becomes a standalone RegionOfInterest when it has no children or when all its children fit into a single region whose total length equals the node's complete representation length. The _check_node_as_roi method implements this check in betterhtmlchunking/tree_regions_system.py, setting node_is_roi = True and creating a region containing only that node's XPath.

How does the ROI system handle nodes that exceed the maximum size limit?

When ROIMaker._process_children encounters a child whose individual length exceeds max_node_repr_length, it excludes the child from the current chunk and adds it to children_to_enqueue. The breadth-first queue in TreeRegionsSystem.start subsequently processes these oversized nodes at a deeper level, breaking them into smaller constituent parts until they fit within the threshold or become atomic elements.

Can I configure whether the system uses HTML length or text length for chunking?

Yes. The repr_length_compared_by parameter accepts ReprLengthComparisionBy.HTML_LENGTH or ReprLengthComparisionBy.TEXT_LENGTH. This setting controls whether TreeRegionsSystem.get_node_repr_length counts raw HTML characters (including tags) or visible text content only, allowing the system to adapt to rendering constraints versus content analysis requirements.

What happens if no single RegionOfInterest satisfies the size constraints?

If the algorithm generates no ROI that meets the size limit—typically when the entire document is smaller than the threshold—the system collapses the whole document into a single RegionOfInterest. As implemented in TreeRegionsSystem.start, this fallback ensures that valid output always exists even when constraints cannot be met through splitting.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}