how-to-guide

How to Use the betterhtmlchunking tree_representation API to Retrieve Node Metadata (text_length and html_length)

February 26, 2026 carlosplanchon/betterhtmlchunking ↗

The DOMTreeRepresentation class exposes node metadata—including text_length and html_length—through the xpaths_metadata dictionary, which maps positional XPaths to NodeMetadata instances.

The betterhtmlchunking library provides a robust tree_representation API that parses HTML into a navigable DOM tree and computes per-node metadata. This metadata includes the character count of visible text and the serialized HTML length, enabling precise content analysis and chunking strategies.

Understanding the DOMTreeRepresentation Class

The DOMTreeRepresentation class, defined in betterhtmlchunking/tree_representation.py, serves as the primary interface for DOM analysis. When initialized with an HTML string, it performs three core operations:

  1. Parses the HTML using BeautifulSoup to create a navigable soup object.
  2. Computes positional XPaths for every element via the compute_xpaths_data method.
  3. Builds a tree representation using treelib and attaches metadata to each node.

This process populates the xpaths_metadata attribute, which stores NodeMetadata objects keyed by their positional XPath strings.

Accessing Node Metadata via xpaths_metadata

The NodeMetadata Data Structure

Each entry in xpaths_metadata is an instance of the NodeMetadata class (lines 58-80 in betterhtmlchunking/tree_representation.py). This dataclass contains:

  • text_length: The character count of the node's rendered text content (excluding HTML tags).
  • html_length: The character count of the node's serialized HTML representation.

These metrics allow you to evaluate content density and make informed decisions about chunking boundaries.

Retrieving Metadata by XPath

To access metadata for a specific node, use its positional XPath as a key in the xpaths_metadata dictionary:

from betterhtmlchunking.tree_representation import DOMTreeRepresentation

html_content = """
<html>
  <body>
    <div id="content">
      <p>Hello <strong>World</strong>!</p>
      <p>Second paragraph with more text content.</p>
    </div>
  </body>
</html>
"""

# Initialize the representation

rep = DOMTreeRepresentation(website_code=html_content)

# Access metadata using the positional XPath

xpath = "/html/body/div[1]/p[2]"
metadata = rep.xpaths_metadata[xpath]

print(f"Text length: {metadata.text_length}")
print(f"HTML length: {metadata.html_length}")

Alternative Access Methods

Via the treelib Tree Node

The DOMTreeRepresentation exposes the underlying treelib.Tree instance through the tree attribute. Each node in this tree stores its NodeMetadata in the data attribute, providing an alternative access path:


# Get the tree node directly

tree_node = rep.tree.get_node("/html/body/div[1]/p[1]")
node_metadata = tree_node.data

# Same object as rep.xpaths_metadata[xpath]

assert node_metadata is rep.xpaths_metadata["/html/body/div[1]/p[1]"]

This approach is useful when traversing the tree structure using treelib methods like children() or subtree().

Practical Code Example

The following complete example demonstrates initializing the representation, listing available XPaths, and retrieving specific node metadata:

from betterhtmlchunking.tree_representation import DOMTreeRepresentation

html_source = """
<!DOCTYPE html>
<html>
  <head><title>Demo Page</title></head>
  <body>
    <article>
      <h1>Main Title</h1>
      <p>This is the <em>first</em> paragraph with some text.</p>
      <p>This is the second paragraph with even more text content to analyze.</p>
    </article>
  </body>
</html>
"""

# Step 1: Create the representation

rep = DOMTreeRepresentation(website_code=html_source)

# Step 2: View all available positional XPaths

print("Available XPaths:")
for xpath in rep.pos_sorted_xpaths:
    print(f"  {xpath}")

# Step 3: Retrieve metadata for specific nodes

target_xpath = "/html/body/article[1]/p[1]"
if target_xpath in rep.xpaths_metadata:
    meta = rep.xpaths_metadata[target_xpath]
    print(f"\nNode: {target_xpath}")
    print(f"  Text length: {meta.text_length}")
    print(f"  HTML length: {meta.html_length}")

# Step 4: Alternative access via tree node

tree_node = rep.tree.get_node(target_xpath)
print(f"\nVia tree node data attribute:")
print(f"  Text length: {tree_node.data.text_length}")

Summary

  • The DOMTreeRepresentation class in betterhtmlchunking/tree_representation.py parses HTML and computes per-node metadata automatically.
  • Node metadata—including text_length and html_length—is stored in NodeMetadata objects accessible via the xpaths_metadata dictionary.
  • Use positional XPaths (e.g., /html/body/div[1]/p[2]) as keys to retrieve specific node metadata.
  • Access metadata alternatively through the data attribute of nodes in the underlying treelib.Tree instance.

Frequently Asked Questions

What is the difference between text_length and html_length in betterhtmlchunking?

The text_length attribute represents the character count of the node's visible text content after stripping all HTML tags, while html_length represents the character count of the node's serialized HTML representation including tags and attributes. For example, a paragraph containing <p>Hello <strong>World</strong></p> would have a text_length of 11 (for "Hello World") but a larger html_length accounting for the markup.

How does DOMTreeRepresentation generate positional XPaths?

The compute_xpaths_data method in betterhtmlchunking/tree_representation.py traverses the BeautifulSoup parse tree and generates positional XPaths (POS-XPaths) by tracking the index of each element among its siblings with the same tag name. These XPaths follow the format /html/body/div[1]/p[2], where the indices indicate the position within the parent container, ensuring every node has a unique, deterministic identifier.

Can I modify node metadata after the tree is created?

The NodeMetadata objects stored in xpaths_metadata are standard Python dataclass instances, so you can technically modify their attributes (such as text_length or html_length) after retrieval. However, these modifications only affect the in-memory object and do not trigger recalculation of the DOM structure. To update the representation after structural changes, you must create a new DOMTreeRepresentation instance with the modified HTML.

Where is the NodeMetadata class defined in the source code?

The NodeMetadata class is defined in betterhtmlchunking/tree_representation.py at approximately lines 58-80. This dataclass encapsulates the metadata for individual DOM nodes, including the node index, text_length, and html_length attributes. The class is imported and used throughout the module to populate the xpaths_metadata dictionary during the tree construction process.

Have a question about this repo?

These articles cover the highlights, but your codebase questions are specific. Give your agent direct access to the source. Share this with your agent to get started:

Share the following with your agent to get started:
curl -s "https://instagit.com/install.md"
Works with
Claude Codex Cursor VS Code OpenClaw Any MCP Client

Maintain an open-source project? Get it listed too →