# How BetterHTMLChunking Preserves Parent-Child Relationships and Hierarchical Context in HTML Chunks > Discover how BetterHTMLChunking maintains hierarchical context in HTML chunks. Learn how it uses XPath and BeautifulSoup to preserve parent-child relationships for better data integrity. - Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking) - Tags: internals - Published: 2026-02-26 --- **BetterHTMLChunking preserves parent-child relationships and hierarchical context in chunks by parsing HTML into a positional XPath-based tree structure using BeautifulSoup, storing nodes in a `treelib.Tree` with explicit parent references, and grouping children into size-bounded regions of interest while maintaining DOM ancestry links.** The `carlosplanchon/betterhtmlchunking` library solves the common problem of flattening HTML documents into chunks without losing structural meaning. By encoding every element's position as a positional XPath and mirroring the DOM hierarchy in a tree data structure, the system ensures that each generated chunk retains enough metadata to reconstruct parent-child relationships and preserve hierarchical context for downstream processing. ## Building a Hierarchical Tree Representation with Positional XPaths ### Parsing HTML and Generating Positional XPaths The foundation of hierarchical preservation begins in [`betterhtmlchunking/tree_representation.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_representation.py). The `DOMTreeRepresentation` class parses raw HTML using **BeautifulSoup**, then walks every element to assign a **positional XPath** (e.g., `/html/body/div[1]/p[2]`). This XPath uniquely encodes the element's depth and sibling index in the DOM. The helper function `get_parent_xpath()` derives ancestry by stripping the last segment from any XPath: ```python # From betterhtmlchunking/tree_representation.py def get_parent_xpath(xpath: str) -> str: """Returns parent xpath from children xpath.""" xpath_list = xpath.split("/") return "/".join(xpath_list[:-1]) ``` ### Constructing the treelib.Tree Structure After generating all XPaths, the code constructs a `treelib.Tree` object that mirrors the DOM hierarchy exactly. The implementation creates a root node with `identifier="root"`, then iterates through every positional XPath to create nodes using `self.tree.create_node()`: ```python # From betterhtmlchunking/tree_representation.py (lines 76-85) self.tree.create_node( tag=pos_xpath, identifier=pos_xpath, parent=parent_xpath, data=node_metadata ) ``` Because each node's `parent` parameter is derived from `get_parent_xpath()`, the resulting tree maintains exact parent-child relationships from the original HTML document. ### Storing Node Metadata with DOM References The `NodeMetadata` dataclass attaches critical context to each tree node, storing `text_length`, `html_length`, and a reference to the original **BeautifulSoup** element. This metadata travels with the tree structure through the entire pipeline, enabling size-based chunking decisions without losing access to the underlying DOM node. ## Generating Regions of Interest While Preserving Hierarchy ### Traversing the Tree Breadth-First The `TreeRegionsSystem` class in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py) manages the chunking process while respecting the hierarchical tree structure. It traverses the `treelib.Tree` breadth-first starting from a configurable root (default `/html`), ensuring that parent nodes are processed before their children. For each node encountered, the system extracts **direct children** using `self.tree_representation.get_children_tag_list(xpath)`. This method returns only immediate descendants in the DOM, not deeper nested nodes, ensuring that parent-child boundaries remain intact during processing. ### Grouping Children into Size-Bounded Chunks The `ROIMaker` class receives the list of child XPaths and groups them into `RegionOfInterest` objects. Each ROI is constrained by `max_node_repr_length`, a parameter that limits the total size of HTML content within a single chunk. The grouping logic ensures that siblings that fit within the size limit remain together in the same chunk, preserving their shared parent context. ### Handling Oversized Nodes with Recursive Processing When a child node exceeds `max_node_repr_length`, the system does not split it arbitrarily. Instead, it **enqueues the node for deeper processing**, making that oversized node the root of its own sub-chunk. This recursive approach guarantees that oversized content becomes the parent of its own ROI while maintaining the surrounding hierarchical context from higher levels. ### Detecting When Parents Become Single ROIs The `_check_node_as_roi()` method implements intelligent boundary detection to determine when a parent node itself should constitute a single chunk. It checks two conditions: 1. **Leaf nodes**: Nodes with no children naturally become individual ROIs. 2. **Aggregated children**: When all children fit within a single region, the parent's XPath is stored as the sole element of the ROI, preserving the hierarchical boundary rather than fragmenting the parent across multiple chunks. ## Maintaining Document Order and Context ### Ordering ROIs by Original DOM Position After generating regions, `order_regions_of_interest_by_pos_xpath()` ensures chunks follow the original document flow. The method builds a mapping from each XPath to its index in the original `pos_xpaths_list` (which reflects the DOM order), then sorts ROIs accordingly. This maintains the **sequential flow** of the document while respecting the tree hierarchy established during processing. ### Accessing Hierarchical Context in Output Each `RegionOfInterest` object exposes metadata that enables downstream reconstruction of parent-child relationships: - `pos_xpath_list`: Contains the XPaths of all elements in the chunk. - `node_is_roi`: Boolean indicating if the chunk represents a complete parent node. - Tree traversal methods: Allow navigation to parent or sibling chunks using the underlying `treelib.Tree`. ## Code Examples ### Building the Tree and Obtaining Parent-Child Mappings ```python from betterhtmlchunking.tree_representation import DOMTreeRepresentation html = "

First

Second

" tree = DOMTreeRepresentation(website_code=html) # The treelib Tree is accessible via `tree.tree` root_children = tree.tree.children('root') print([node.identifier for node in root_children]) # → ['/html'] ``` ### Getting the XPath Depth for Ordering ```python from betterhtmlchunking.tree_representation import get_xpath_depth xpath = "/html/body/div[1]/p[2]" depth = get_xpath_depth(xpath) # returns 4 (root → html → body → div → p) ``` ### Creating ROIs with a Size Limit ```python from betterhtmlchunking.tree_regions_system import TreeRegionsSystem, ReprLengthComparisionBy system = TreeRegionsSystem( tree_representation=tree, max_node_repr_length=200, repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH ) # `system.sorted_roi_by_pos_xpath` holds the final, ordered chunks for idx, roi in system.sorted_roi_by_pos_xpath.items(): print(f"Chunk {idx}:") for xpath in roi.pos_xpath_list: print(f" {xpath}") ``` ### Inspecting Hierarchical Context in a Single ROI ```python roi = next(iter(system.sorted_roi_by_pos_xpath.values())) # first ROI print("Is this ROI a whole node?", roi.node_is_roi) # True if the parent itself is the ROI print("Contained XPaths:", roi.pos_xpath_list) # Child XPaths that belong to this chunk ``` ## Summary - **Positional XPaths encode hierarchy**: Every HTML element receives a unique XPath (`/html/body/div[1]/p[2]`) that captures its exact position in the DOM tree, enabling precise parent-child relationship tracking. - **treelib.Tree mirrors the DOM**: The library constructs a complete tree data structure where each node's parent is derived from its XPath, ensuring the hierarchical structure remains accessible throughout the chunking process. - **Size-bounded grouping respects boundaries**: The `ROIMaker` groups sibling elements into chunks limited by `max_node_repr_length`, while oversized nodes are recursively processed as new sub-roots rather than being arbitrarily split. - **Parent aggregation preserves context**: When all children of a parent fit within a single region, the parent itself becomes the ROI, maintaining the hierarchical boundary instead of fragmenting the structure across multiple chunks. - **Document order is maintained**: Final chunks are sorted by their original DOM position using `order_regions_of_interest_by_pos_xpath()`, ensuring sequential flow while respecting the tree hierarchy. ## Frequently Asked Questions ### How does BetterHTMLChunking handle deeply nested HTML structures? BetterHTMLChunking handles deeply nested structures by encoding every element's depth into its positional XPath and storing the entire hierarchy in a `treelib.Tree`. When processing nested content, the system traverses breadth-first and treats oversized nested nodes as new roots for recursive processing. This ensures that even deeply nested elements maintain their ancestral relationships while being chunked appropriately according to `max_node_repr_length`. ### What happens if a single HTML element exceeds the maximum chunk size? When a single element exceeds `max_node_repr_length`, the `ROIMaker` does not split the element arbitrarily. Instead, it enqueues that node for deeper processing, making it the root of its own sub-chunk. This recursive approach ensures that oversized content becomes a parent ROI containing its own children, preserving the hierarchical context rather than breaking the element across chunk boundaries. ### Can BetterHTMLChunking reconstruct the original document structure from chunks? Yes, BetterHTMLChunking retains sufficient metadata to reconstruct the original structure. Each `RegionOfInterest` stores `pos_xpath_list` containing the exact XPaths of included elements, and the underlying `treelib.Tree` maintains parent-child relationships. Because XPaths encode the original DOM position and the `node_is_roi` flag indicates when a parent represents a complete hierarchical boundary, downstream systems can reassemble chunks into the original tree structure or reason about the context of each fragment. ### How does the library ensure chunks appear in the correct reading order? The library ensures correct reading order through the `order_regions_of_interest_by_pos_xpath()` method in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py). This method maps each XPath to its original index in the `pos_xpaths_list` (which reflects the DOM's natural order) and sorts the ROIs accordingly. This sorting happens after the hierarchical grouping is complete, ensuring that the final chunks respect both the tree structure and the sequential flow of the original document.