# How the RegionOfInterest (ROI) System Groups DOM Nodes into Semantically Coherent Chunks

> Discover how the RegionOfInterest system groups DOM nodes into semantically coherent chunks. Learn its efficient tree traversal and chunking strategy.

- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: internals
- Published: 2026-02-26

---

**The RegionOfInterest system groups DOM nodes into semantically coherent chunks by traversing the document tree breadth-first, packing siblings into size-constrained regions, and promoting parent elements to single chunks when their children fit entirely within the threshold.**

The **betterhtmlchunking** library solves the problem of splitting complex HTML documents into manageable, contextually meaningful segments. At its heart, the **RegionOfInterest (ROI) system** employs a three-stage pipeline that walks the DOM tree, evaluates node representations, and assembles logical chunks without fragmenting semantic boundaries. This mechanism ensures that related content—such as complete paragraphs or list items—remains grouped even when enforcing strict size limits for downstream processing like summarization or translation.

## The Three-Stage ROI Creation Pipeline

The ROI creation process operates through tightly coupled stages defined in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py), utilizing a breadth-first strategy to maintain document order and sibling relationships.

### Stage 1: Breadth-First Tree Traversal via `TreeRegionsSystem.start`

The traversal engine initializes with a FIFO queue (`queue.Queue`) that manages XPaths requiring examination. According to the source code in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py), the algorithm begins by selecting the root node (`/html` if present, otherwise the first available XPath) and placing it on the queue.

While the queue contains nodes, the system:

1. Pops the next XPath and retrieves its direct children via `get_children_tag_list`
2. Delegates child processing to the **`ROIMaker`** class
3. Collects all `RegionOfInterest` objects returned by the maker
4. Re-queues any children too large to fit into current chunks for deeper processing

After the loop completes, the raw ROI list undergoes sorting by document order using `order_regions_of_interest_by_pos_xpath`. If no generated ROI satisfies the size limit, the system collapses the entire document into a single RegionOfInterest.

### Stage 2: Size-Constrained Chunk Packing in `ROIMaker._process_children`

For each set of siblings retrieved during traversal, **`ROIMaker._process_children`** evaluates whether they can join the current chunk without exceeding `max_node_repr_length`. The method queries node representation lengths via `TreeRegionsSystem.get_node_repr_length`, which returns either text length or HTML length based on the `repr_length_compared_by` configuration (`ReprLengthComparisionBy.TEXT_LENGTH` or `ReprLengthComparisionBy.HTML_LENGTH`).

The packing logic enforces these rules:

- If a single child exceeds the limit, it bypasses the current chunk and enters `children_to_enqueue` for deeper decomposition
- When adding a child would cross the threshold, the current ROI closes and a new one initializes
- After processing all children, any leftover region merges with the previous chunk (if one exists) or stands as the sole region

This greedy packing strategy ensures that sequential siblings appearing together in markup—such as consecutive `<li>` elements—remain grouped when their combined size permits.

### Stage 3: Semantic Node Promotion via `_check_node_as_roi`

The final stage determines whether a parent node should collapse into a single RegionOfInterest. In `ROIMaker._check_node_as_roi`, the system evaluates if a node has **no children** or if **all children occupy a single region whose total length equals the node's full representation length**.

When these conditions meet, the algorithm sets `node_is_roi = True` and creates a `RegionOfInterest` containing only the parent node's XPath. This promotion preserves higher-level semantics by keeping intact structures—such as a `<section>` containing only small inline elements—as unified chunks rather than fragmenting them across multiple regions.

## How Semantic Coherence Is Achieved

The ROI system maintains semantic integrity through four specific design decisions:

- **Size-based packing with configurable limits**: By enforcing `max_node_repr_length`, the system prevents oversized chunks that would break rendering pipelines while keeping related content together when possible.

- **Breadth-first sibling processing**: Examining nodes level-by-level ensures that semantically related siblings (e.g., list items, table cells) are considered for grouping before the algorithm descends into their children.

- **Parent-node promotion**: When a subtree fits entirely within constraints, the `_check_node_as_roi` method collapses it into a single chunk, preserving the document's hierarchical meaning rather than splitting a cohesive element arbitrarily.

- **Flexible length calculation**: The `repr_length_compared_by` parameter allows callers to choose between `HTML_LENGTH` (character count including tags) or `TEXT_LENGTH` (visible content only), adapting chunking behavior for use cases ranging from safe HTML rendering to text summarization.

## Practical Implementation Examples

The following examples demonstrate how to configure and execute the ROI system using the high-level `DomRepresentation` façade and the low-level `TreeRegionsSystem` directly.

```python
from betterhtmlchunking.main import DomRepresentation
from betterhtmlchunking.tree_regions_system import ReprLengthComparisionBy

# Example HTML input

html = """
<article>
  <h1>Title</h1>
  <p>This is a short paragraph.</p>
  <p>Another paragraph with <strong>bold</strong> text.</p>
  <img src="pic.jpg"/>
</article>
"""

# Build the full pipeline – each chunk will be ≤ 150 characters of HTML

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=150,
    website_code=html,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH,
)

# Run the pipeline

dom.start(verbose=True, maximal_verbose=False)

# Access the generated ROIs (chunks)

for idx, roi in dom.tree_regions_system.sorted_roi_by_pos_xpath.items():
    print(f"Chunk {idx}:")
    print("  XPaths:", roi.pos_xpath_list)
    print("  HTML length:", roi.repr_length)

```

```python

# Direct low‑level usage without the full DomRepresentation wrapper

from betterhtmlchunking.tree_representation import DOMTreeRepresentation
from betterhtmlchunking.tree_regions_system import TreeRegionsSystem, ReprLengthComparisionBy

tree_repr = DOMTreeRepresentation(website_code=html)
tree_repr.recompute_representation()

roi_system = TreeRegionsSystem(
    tree_representation=tree_repr,
    max_node_repr_length=200,                     # limit per chunk

    repr_length_compared_by=ReprLengthComparisionBy.TEXT_LENGTH,
)

# After construction the ROI list is ready

for i, roi in roi_system.sorted_roi_by_pos_xpath.items():
    print(f"ROI {i} – {len(roi.pos_xpath_list)} nodes, text length {roi.repr_length}")

```

## Core Source Files and Architecture

Understanding the ROI system requires familiarity with these key components in the `betterhtmlchunking` repository:

- **[`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py)**: Contains the core ROI creation logic, including `TreeRegionsSystem`, `ROIMaker`, and the document ordering utilities.

- **[`betterhtmlchunking/tree_representation.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_representation.py)**: Builds a treelib tree from BeautifulSoup and provides node metadata such as `text_length` and `html_length` used during size calculations.

- **[`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py)**: Implements the high-level `DomRepresentation` façade that orchestrates the tree representation, ROI system, and rendering pipeline.

- **[`betterhtmlchunking/utils.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/utils.py)**: Provides preprocessing utilities that remove unwanted tags (such as `<script>` and `<style>`) before ROI processing begins.

- **[`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py)**: Consumes the final ROI list to produce HTML or text chunks suitable for downstream applications like translation services or content summarization.

## Summary

- The **RegionOfInterest system** in betterhtmlchunking uses a **breadth-first traversal** (`TreeRegionsSystem.start`) to process DOM nodes in document order while maintaining a queue for oversized elements requiring deeper inspection.

- **Chunk creation** occurs in `ROIMaker._process_children`, which greedily packs sibling nodes into regions until reaching `max_node_repr_length`, using either HTML or text length calculations based on the `repr_length_compared_by` setting.

- **Semantic promotion** via `_check_node_as_roi` collapses entire subtrees into single chunks when their children fit within constraints, preventing fragmentation of cohesive elements like paragraphs or sections.

- The system guarantees **semantic coherence** by processing siblings together, respecting size limits, and preserving parent-node integrity when possible.

## Frequently Asked Questions

### What triggers a node to be treated as its own RegionOfInterest?

A node becomes a standalone RegionOfInterest when it has no children or when all its children fit into a single region whose total length equals the node's complete representation length. The `_check_node_as_roi` method implements this check in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py), setting `node_is_roi = True` and creating a region containing only that node's XPath.

### How does the ROI system handle nodes that exceed the maximum size limit?

When `ROIMaker._process_children` encounters a child whose individual length exceeds `max_node_repr_length`, it excludes the child from the current chunk and adds it to `children_to_enqueue`. The breadth-first queue in `TreeRegionsSystem.start` subsequently processes these oversized nodes at a deeper level, breaking them into smaller constituent parts until they fit within the threshold or become atomic elements.

### Can I configure whether the system uses HTML length or text length for chunking?

Yes. The `repr_length_compared_by` parameter accepts `ReprLengthComparisionBy.HTML_LENGTH` or `ReprLengthComparisionBy.TEXT_LENGTH`. This setting controls whether `TreeRegionsSystem.get_node_repr_length` counts raw HTML characters (including tags) or visible text content only, allowing the system to adapt to rendering constraints versus content analysis requirements.

### What happens if no single RegionOfInterest satisfies the size constraints?

If the algorithm generates no ROI that meets the size limit—typically when the entire document is smaller than the threshold—the system collapses the whole document into a single RegionOfInterest. As implemented in `TreeRegionsSystem.start`, this fallback ensures that valid output always exists even when constraints cannot be met through splitting.