how-to-guide

How to Access Positional XPaths to Trace HTML Chunks Back to Source in betterhtmlchunking

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

You can access positional XPaths through RegionOfInterest.pos_xpath_list after processing HTML with DomRepresentation, which maps each chunk to its exact source location using indexed paths like /html/body/div[2]/p[1].

The betterhtmlchunking library generates intelligent chunks from HTML documents while preserving traceability to the original markup. When you need to access positional XPaths to verify chunk origins or debug content extraction, the library exposes these paths through its DOM tree representation and region detection system.

Understanding Positional XPaths in betterhtmlchunking

betterhtmlchunking builds a full DOM-tree representation of the input HTML. During this process, every element receives a positional XPath—a path containing the exact index of each sibling (e.g., /html/body/div[2]/p[1]).

The library stores these XPaths in two primary locations:

  • DOMTreeRepresentation.pos_xpaths_list – A flat list of all positional XPaths generated from the document (e.g., ["/html", "/html/body", "/html/body/div[1]", …])
  • RegionOfInterest.pos_xpath_list – The subset of XPaths that compose a single chunk (ROI) (e.g., ["/html/body/div[2]/h1", "/html/body/div[2]/p[1]"])

When the TreeRegionsSystem finishes processing, it creates a dictionary TreeRegionsSystem.sorted_roi_by_pos_xpath that maps a numeric chunk index to a RegionOfInterest instance. Each ROI contains the exact positional XPaths that generated that chunk.

Accessing XPaths from the DOM Tree Representation

Retrieving All XPaths from DOMTreeRepresentation

The DOMTreeRepresentation class in betterhtmlchunking/tree_representation.py computes positional XPaths during initialization. You can access the complete list via the pos_xpaths_list attribute:

from betterhtmlchunking.main import DomRepresentation

dom = DomRepresentation(
    MAX_NODE_REPR_LENGTH=2000,
    website_code="<html><body><div><p>Text</p></div></body></html>",
)
dom.start()

# Access all positional XPaths in the document

all_xpaths = dom.tree_representation.pos_xpaths_list
print(all_xpaths)

# Output: ['/html', '/html/body', '/html/body/div', '/html/body/div/p']

Mapping XPaths to BeautifulSoup Elements

To trace an XPath back to its original HTML element, use the xpaths_metadata dictionary. Each entry maps an XPath to a NodeMetadata object containing the bs4_elem (BeautifulSoup Tag):


# Get the BeautifulSoup Tag for a specific XPath

xpath = "/html/body/div[1]/p[2]"
metadata = dom.tree_representation.xpaths_metadata[xpath]
original_tag = metadata.bs4_elem

print(original_tag.prettify())

This gives you the exact source snippet that contributed to the chunk.

Tracing Chunks via RegionOfInterest

Accessing the Sorted ROI Dictionary

The TreeRegionsSystem class in betterhtmlchunking/tree_regions_system.py groups DOM nodes into chunks (Regions of Interest). After processing, access sorted_roi_by_pos_xpath to map chunk indices to their source XPaths:


# Access the dictionary mapping chunk index to RegionOfInterest

roi_dict = dom.tree_regions_system.sorted_roi_by_pos_xpath

# Get chunk 0's source XPaths

first_chunk_xpaths = roi_dict[0].pos_xpath_list
print(first_chunk_xpaths)

Extracting XPaths for Specific Chunks

Iterate over the sorted dictionary to trace every chunk back to its origin:

for idx, roi in dom.tree_regions_system.sorted_roi_by_pos_xpath.items():
    print(f"--- Chunk {idx} ---")
    for xpath in roi.pos_xpath_list:
        print(xpath)

Example output:


--- Chunk 0 ---
/html/body/div[1]/h1
--- Chunk 1 ---
/html/body/div[1]/p[1]
/html/body/div[1]/p[2]
--- Chunk 2 ---
/html/body/div[2]/ul/li[1]
/html/body/div[2]/ul/li[2]

Practical Code Examples for XPath Retrieval

Function to Get Source XPaths by Chunk Index

def get_chunk_source_xpaths(dom, chunk_idx: int) -> list[str]:
    """Return the list of positional XPaths that built the requested chunk."""
    roi = dom.tree_regions_system.sorted_roi_by_pos_xpath.get(chunk_idx)
    if not roi:
        raise IndexError(f"Chunk {chunk_idx} does not exist")
    return roi.pos_xpath_list

# Usage

paths = get_chunk_source_xpaths(dom, 2)
print("\n".join(paths))

Mapping Chunks to Original HTML Snippets

def chunk_to_html(dom, chunk_idx: int) -> str:
    """Concatenate the HTML of all nodes that belong to the chunk."""
    xpaths = get_chunk_source_xpaths(dom, chunk_idx)
    snippets = [
        dom.tree_representation.xpaths_metadata[x].bs4_elem.prettify()
        for x in xpaths
    ]
    return "\n".join(snippets)

print(chunk_to_html(dom, 0))

CLI Verbose Output for Debugging

Use the command-line interface with verbose flags to inspect positional XPaths without writing code:


# Run the CLI with maximal verbose to see the internal XPaths

cat page.html | betterhtmlchunking --max-length 1500 --maximal-verbose

The verbose log prints lines like:


ROI 0 → pos_xpath_list=['/html/body/div[1]/h1']
ROI 1 → pos_xpath_list=['/html/body/div[1]/p[1]', '/html/body/div[1]/p[2]']

Key Implementation Files

File Role Link
betterhtmlchunking/tree_representation.py Builds the DOM tree, computes and stores positional XPaths (pos_xpaths_list, xpaths_metadata). tree_representation.py
betterhtmlchunking/tree_regions_system.py Detects Regions-of-Interest (ROIs) and groups the XPaths that belong to each chunk (RegionOfInterest.pos_xpath_list, sorted_roi_by_pos_xpath). tree_regions_system.py
betterhtmlchunking/utils.py Helper for filtering XPaths (e.g., removing unwanted tags). utils.py
README.md High-level usage examples and explanation of chunking workflow. README.md

Summary

  • Positional XPaths in betterhtmlchunking use indexed notation (e.g., /html/body/div[2]/p[1]) to pinpoint exact element locations in the source HTML.
  • Access the complete list of document XPaths via DOMTreeRepresentation.pos_xpaths_list or map individual XPaths to BeautifulSoup elements using xpaths_metadata.
  • Trace specific chunks through TreeRegionsSystem.sorted_roi_by_pos_xpath, where each RegionOfInterest stores its source XPaths in pos_xpath_list.
  • Retrieve original HTML snippets by combining pos_xpath_list with xpaths_metadata to access the underlying bs4_elem objects.

Frequently Asked Questions

What is a positional XPath in betterhtmlchunking?

A positional XPath is a string representation of an element's location in the HTML DOM that includes numeric indices for sibling elements (e.g., /html/body/div[2]/p[1]). According to the betterhtmlchunking source code in tree_representation.py, these paths are generated by the get_pos_xpath_from_bs4_elem() function which walks the BeautifulSoup parent chain and counts siblings to create deterministic, index-based paths.

How do I get the original HTML element from a chunk's XPath?

You can retrieve the original BeautifulSoup Tag object by accessing the xpaths_metadata dictionary stored in DOMTreeRepresentation. Each XPath maps to a NodeMetadata object containing the bs4_elem attribute. For example: dom.tree_representation.xpaths_metadata["/html/body/div[1]/p[2]"].bs4_elem returns the original HTML element, which you can then render using .prettify() or process further.

Can I access positional XPaths without using the CLI?

Yes, positional XPaths are fully accessible through the Python API without invoking the command-line interface. After instantiating DomRepresentation and calling .start(), you can access dom.tree_representation.pos_xpaths_list for all document XPaths or dom.tree_regions_system.sorted_roi_by_pos_xpath for chunk-specific XPaths. The CLI's --maximal-verbose flag merely provides a convenient way to view these same values in terminal output.

Where are positional XPaths stored in the source code?

Positional XPaths are primarily stored in two locations within the betterhtmlchunking codebase. In betterhtmlchunking/tree_representation.py, the DOMTreeRepresentation class maintains pos_xpaths_list (a flat list of all paths) and xpaths_metadata (a dictionary mapping paths to element metadata). In betterhtmlchunking/tree_regions_system.py, the RegionOfInterest class stores pos_xpath_list containing only the XPaths belonging to that specific chunk, while TreeRegionsSystem organizes these into sorted_roi_by_pos_xpath.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →