How BetterHTMLChunking Preserves Parent-Child Relationships and Hierarchical Context in HTML Chunks

BetterHTMLChunking preserves parent-child relationships and hierarchical context in chunks by parsing HTML into a positional XPath-based tree structure using BeautifulSoup, storing nodes in a treelib.Tree with explicit parent references, and grouping children into size-bounded regions of interest while maintaining DOM ancestry links.

The carlosplanchon/betterhtmlchunking library solves the common problem of flattening HTML documents into chunks without losing structural meaning. By encoding every element's position as a positional XPath and mirroring the DOM hierarchy in a tree data structure, the system ensures that each generated chunk retains enough metadata to reconstruct parent-child relationships and preserve hierarchical context for downstream processing.

Building a Hierarchical Tree Representation with Positional XPaths

Parsing HTML and Generating Positional XPaths

The foundation of hierarchical preservation begins in betterhtmlchunking/tree_representation.py. The DOMTreeRepresentation class parses raw HTML using BeautifulSoup, then walks every element to assign a positional XPath (e.g., /html/body/div[1]/p[2]). This XPath uniquely encodes the element's depth and sibling index in the DOM.

The helper function get_parent_xpath() derives ancestry by stripping the last segment from any XPath:

# From betterhtmlchunking/tree_representation.py

def get_parent_xpath(xpath: str) -> str:
    """Returns parent xpath from children xpath."""
    xpath_list = xpath.split("/")
    return "/".join(xpath_list[:-1])

Constructing the treelib.Tree Structure

After generating all XPaths, the code constructs a treelib.Tree object that mirrors the DOM hierarchy exactly. The implementation creates a root node with identifier="root", then iterates through every positional XPath to create nodes using self.tree.create_node():

# From betterhtmlchunking/tree_representation.py (lines 76-85)

self.tree.create_node(
    tag=pos_xpath,
    identifier=pos_xpath,
    parent=parent_xpath,
    data=node_metadata
)

Because each node's parent parameter is derived from get_parent_xpath(), the resulting tree maintains exact parent-child relationships from the original HTML document.

Storing Node Metadata with DOM References

The NodeMetadata dataclass attaches critical context to each tree node, storing text_length, html_length, and a reference to the original BeautifulSoup element. This metadata travels with the tree structure through the entire pipeline, enabling size-based chunking decisions without losing access to the underlying DOM node.

Generating Regions of Interest While Preserving Hierarchy

Traversing the Tree Breadth-First

The TreeRegionsSystem class in betterhtmlchunking/tree_regions_system.py manages the chunking process while respecting the hierarchical tree structure. It traverses the treelib.Tree breadth-first starting from a configurable root (default /html), ensuring that parent nodes are processed before their children.

For each node encountered, the system extracts direct children using self.tree_representation.get_children_tag_list(xpath). This method returns only immediate descendants in the DOM, not deeper nested nodes, ensuring that parent-child boundaries remain intact during processing.

Grouping Children into Size-Bounded Chunks

The ROIMaker class receives the list of child XPaths and groups them into RegionOfInterest objects. Each ROI is constrained by max_node_repr_length, a parameter that limits the total size of HTML content within a single chunk. The grouping logic ensures that siblings that fit within the size limit remain together in the same chunk, preserving their shared parent context.

Handling Oversized Nodes with Recursive Processing

When a child node exceeds max_node_repr_length, the system does not split it arbitrarily. Instead, it enqueues the node for deeper processing, making that oversized node the root of its own sub-chunk. This recursive approach guarantees that oversized content becomes the parent of its own ROI while maintaining the surrounding hierarchical context from higher levels.

Detecting When Parents Become Single ROIs

The _check_node_as_roi() method implements intelligent boundary detection to determine when a parent node itself should constitute a single chunk. It checks two conditions:

1. Leaf nodes: Nodes with no children naturally become individual ROIs.
2. Aggregated children: When all children fit within a single region, the parent's XPath is stored as the sole element of the ROI, preserving the hierarchical boundary rather than fragmenting the parent across multiple chunks.

Maintaining Document Order and Context

Ordering ROIs by Original DOM Position

After generating regions, order_regions_of_interest_by_pos_xpath() ensures chunks follow the original document flow. The method builds a mapping from each XPath to its index in the original pos_xpaths_list (which reflects the DOM order), then sorts ROIs accordingly. This maintains the sequential flow of the document while respecting the tree hierarchy established during processing.

Accessing Hierarchical Context in Output

Each RegionOfInterest object exposes metadata that enables downstream reconstruction of parent-child relationships:

• pos_xpath_list: Contains the XPaths of all elements in the chunk.
• node_is_roi: Boolean indicating if the chunk represents a complete parent node.
• Tree traversal methods: Allow navigation to parent or sibling chunks using the underlying treelib.Tree.

Code Examples

Building the Tree and Obtaining Parent-Child Mappings

from betterhtmlchunking.tree_representation import DOMTreeRepresentation

html = "<html><body><div><p>First</p><p>Second</p></div></body></html>"
tree = DOMTreeRepresentation(website_code=html)

# The treelib Tree is accessible via `tree.tree`

root_children = tree.tree.children('root')
print([node.identifier for node in root_children])

# → ['/html']

Getting the XPath Depth for Ordering

from betterhtmlchunking.tree_representation import get_xpath_depth

xpath = "/html/body/div[1]/p[2]"
depth = get_xpath_depth(xpath)   # returns 4 (root → html → body → div → p)

Creating ROIs with a Size Limit

from betterhtmlchunking.tree_regions_system import TreeRegionsSystem, ReprLengthComparisionBy

system = TreeRegionsSystem(
    tree_representation=tree,
    max_node_repr_length=200,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH
)

# `system.sorted_roi_by_pos_xpath` holds the final, ordered chunks

for idx, roi in system.sorted_roi_by_pos_xpath.items():
    print(f"Chunk {idx}:")
    for xpath in roi.pos_xpath_list:
        print(f"  {xpath}")

Inspecting Hierarchical Context in a Single ROI

roi = next(iter(system.sorted_roi_by_pos_xpath.values()))  # first ROI

print("Is this ROI a whole node?", roi.node_is_roi)  # True if the parent itself is the ROI

print("Contained XPaths:", roi.pos_xpath_list)       # Child XPaths that belong to this chunk

Summary

• Positional XPaths encode hierarchy: Every HTML element receives a unique XPath (/html/body/div[1]/p[2]) that captures its exact position in the DOM tree, enabling precise parent-child relationship tracking.

• treelib.Tree mirrors the DOM: The library constructs a complete tree data structure where each node's parent is derived from its XPath, ensuring the hierarchical structure remains accessible throughout the chunking process.

• Size-bounded grouping respects boundaries: The ROIMaker groups sibling elements into chunks limited by max_node_repr_length, while oversized nodes are recursively processed as new sub-roots rather than being arbitrarily split.

• Parent aggregation preserves context: When all children of a parent fit within a single region, the parent itself becomes the ROI, maintaining the hierarchical boundary instead of fragmenting the structure across multiple chunks.

• Document order is maintained: Final chunks are sorted by their original DOM position using order_regions_of_interest_by_pos_xpath(), ensuring sequential flow while respecting the tree hierarchy.

Frequently Asked Questions

How does BetterHTMLChunking handle deeply nested HTML structures?

BetterHTMLChunking handles deeply nested structures by encoding every element's depth into its positional XPath and storing the entire hierarchy in a treelib.Tree. When processing nested content, the system traverses breadth-first and treats oversized nested nodes as new roots for recursive processing. This ensures that even deeply nested elements maintain their ancestral relationships while being chunked appropriately according to max_node_repr_length.

What happens if a single HTML element exceeds the maximum chunk size?

When a single element exceeds max_node_repr_length, the ROIMaker does not split the element arbitrarily. Instead, it enqueues that node for deeper processing, making it the root of its own sub-chunk. This recursive approach ensures that oversized content becomes a parent ROI containing its own children, preserving the hierarchical context rather than breaking the element across chunk boundaries.

Can BetterHTMLChunking reconstruct the original document structure from chunks?

Yes, BetterHTMLChunking retains sufficient metadata to reconstruct the original structure. Each RegionOfInterest stores pos_xpath_list containing the exact XPaths of included elements, and the underlying treelib.Tree maintains parent-child relationships. Because XPaths encode the original DOM position and the node_is_roi flag indicates when a parent represents a complete hierarchical boundary, downstream systems can reassemble chunks into the original tree structure or reason about the context of each fragment.

How does the library ensure chunks appear in the correct reading order?

The library ensures correct reading order through the order_regions_of_interest_by_pos_xpath() method in betterhtmlchunking/tree_regions_system.py. This method maps each XPath to its original index in the pos_xpaths_list (which reflects the DOM's natural order) and sorts the ROIs accordingly. This sorting happens after the hierarchical grouping is complete, ensuring that the final chunks respect both the tree structure and the sequential flow of the original document.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}