How to Combine betterhtmlchunking with LangChain for Hybrid RAG Pipelines

You can combine betterhtmlchunking with LangChain by wrapping each Region of Interest (ROI) from the HTML parser into a LangChain Document object, enabling structure-aware chunking with XPath provenance for hybrid retrieval-augmented generation workflows.

The carlosplanchon/betterhtmlchunking library parses HTML documents into semantically coherent chunks while preserving structural metadata. When building hybrid RAG pipelines with LangChain, you need to bridge these DOM-aware chunks with LangChain's expected Document format to enable vector storage, retrieval, and downstream LLM consumption.

Understanding the betterhtmlchunking Architecture

The library processes HTML through a three-stage pipeline defined across several core modules. Understanding these components helps you correctly extract and format chunks for LangChain integration.

DOM Representation and ROI Detection

The DomRepresentation class in betterhtmlchunking/main.py serves as the primary facade. It orchestrates the parsing of HTML into a DOM tree and computes Regions of Interest (ROIs)—groupings of nodes that respect your defined size limits while maintaining structural boundaries.

from betterhtmlchunking import DomRepresentation, ReprLengthComparisionBy

dom_repr = DomRepresentation(
    MAX_NODE_REPR_LENGTH=2000,                # size limit per ROI

    website_code=html_source,
    repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH
)
dom_repr.start()

The DomRepresentation.start() method triggers the pipeline stages defined in betterhtmlchunking/tree_regions_system.py, where the TreeRegionsSystem and ROIMaker classes implement the core algorithm for grouping nodes into size-constrained regions.

Rendering System

Once ROIs are identified, the RenderSystem class in betterhtmlchunking/render_system.py produces two parallel dictionaries:

• html_render_roi[idx] – The full HTML representation of the region
• text_render_roi[idx] – The plain-text extraction of the same region

These dictionaries map region indices to their respective content formats, enabling you to provide both structured HTML and clean text to LangChain.

Integrating with LangChain Documents

LangChain's RAG workflows expect objects containing page_content (text for the LLM) and metadata (arbitrary key-value pairs for provenance). You bridge betterhtmlchunking to LangChain by mapping each ROI to a Document instance.

Conversion Function

The following function converts the output of DomRepresentation into a list of LangChain Document objects:

from betterhtmlchunking import DomRepresentation, ReprLengthComparisionBy
from langchain.docstore.document import Document

def html_to_langchain_docs(
    html_source: str,
    max_len: int = 2000,
    compare_by: ReprLengthComparisionBy = ReprLengthComparisionBy.HTML_LENGTH,
) -> list[Document]:
    """Convert raw HTML into LangChain Document objects using betterhtmlchunking."""
    # Build DOM representation and run the pipeline

    dom = DomRepresentation(
        MAX_NODE_REPR_LENGTH=max_len,
        website_code=html_source,
        repr_length_compared_by=compare_by,
    )
    dom.start()

    # Pull rendered chunks

    html_chunks = dom.render_system.html_render_roi
    text_chunks = dom.render_system.text_render_roi
    rois = dom.tree_regions_system.sorted_roi_by_pos_xpath

    # Wrap each ROI in a LangChain Document

    docs: list[Document] = []
    for idx, _ in html_chunks.items():
        docs.append(
            Document(
                page_content=text_chunks[idx],
                metadata={
                    "chunk_index": idx,
                    "html": html_chunks[idx],
                    "xpaths": rois[idx].pos_xpath_list,
                },
            )
        )
    return docs

This implementation preserves the XPath provenance in metadata["xpaths"], stores the original HTML in metadata["html"], and uses the plain-text version as page_content for embedding and LLM consumption.

Building the Hybrid RAG Pipeline

With the conversion function in place, you can construct a complete hybrid RAG pipeline that leverages structure-aware chunking for retrieval and optional token-level splitting for generation.

Vector Store Indexing

Index the documents using your preferred vector store and embedding model:

from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS

# Load and chunk HTML

with open("documentation.html", "r", encoding="utf-8") as f:
    html_raw = f.read()

docs = html_to_langchain_docs(html_raw, max_len=1500)

# Create vector store

embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})

Retrieval and Optional Token Splitting

Retrieve relevant chunks and optionally apply secondary token-based splitting if the structural chunks exceed your LLM's context window:

from langchain.text_splitter import RecursiveCharacterTextSplitter

query = "What are the deployment requirements for the API?"
relevant_docs = retriever.get_relevant_documents(query)

# Hybrid approach: split only if necessary

splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=200)
final_chunks = []
for doc in relevant_docs:
    if len(doc.page_content) > 1024:
        final_chunks.extend(splitter.split_documents([doc]))
    else:
        final_chunks.append(doc)

End-to-End Integration Example

Combine all components into a complete retrieval chain:

from langchain.llms import OpenAI
from langchain.chains import RetrievalQA

# Initialize LLM and QA chain

llm = OpenAI()
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=retriever,
    return_source_documents=True,
)

# Execute query

result = qa_chain.run("Explain the authentication flow")
print(result["result"])

# Access provenance

for source in result["source_documents"]:
    print(f"Chunk {source.metadata['chunk_index']}: {source.metadata['xpaths']}")

Key Source Files and Implementation Details

The following files in the carlosplanchon/betterhtmlchunking repository implement the core functionality referenced above:

• betterhtmlchunking/main.py – Contains the DomRepresentation class that serves as the primary API entry point, orchestrating the pipeline through its start() method and __attrs_post_init__ initialization.

• betterhtmlchunking/tree_representation.py – Implements the DOM tree construction, node length computation (both plain-text and HTML), and XPath-to-node mapping.

• betterhtmlchunking/tree_regions_system.py – Houses the TreeRegionsSystem and ROIMaker classes that implement the region detection algorithm, grouping nodes into size-constrained ROIs while preserving structural boundaries.

• betterhtmlchunking/render_system.py – Provides the RenderSystem class that generates parallel HTML and plain-text dictionaries for each ROI index.

• betterhtmlchunking/utils.py – Contains helper functions for filtering unwanted tags before chunking.

• betterhtmlchunking/__init__.py – Exports the public API including DomRepresentation and ReprLengthComparisionBy.

Summary

• betterhtmlchunking parses HTML into structure-aware Regions of Interest (ROIs) that respect size limits while preserving DOM boundaries and XPath provenance.
• You can integrate these ROIs with LangChain by wrapping each region in a Document object, using the plain-text content for page_content and storing HTML snippets, XPaths, and indices in metadata.
• This approach enables hybrid RAG pipelines that combine structural chunking for semantic retrieval with optional token-based splitting for generation, ensuring precise provenance traceability back to specific HTML elements.

Frequently Asked Questions

What is the primary advantage of using betterhtmlchunking over standard text splitters?

Standard text splitters like LangChain's RecursiveCharacterTextSplitter treat HTML as raw text, often splitting across semantic boundaries like paragraphs or list items. betterhtmlchunking respects the DOM structure, ensuring that related elements stay together in the same chunk and preserving XPath provenance for traceability back to the source HTML.

How does the hybrid approach handle oversized HTML regions?

If a structural ROI from betterhtmlchunking exceeds your LLM's token limit, you can apply a secondary token-based splitter (such as RecursiveCharacterTextSplitter) only to those specific chunks. This "structure first, token-size second" approach maintains semantic coherence while ensuring content fits within context windows.

Can I use betterhtmlchunking with other LLM frameworks besides LangChain?

Yes. The library outputs standard Python dictionaries containing HTML snippets, plain text, and XPath lists. You can adapt the conversion logic to create document objects for LlamaIndex, Haystack, or custom frameworks by mapping the ROI fields to the respective framework's document schema.

What metadata is available for each chunk in the LangChain Document?

Each Document object contains a metadata dictionary with three key fields: chunk_index (the integer ID of the ROI), html (the full HTML snippet of the region), and xpaths (a list of XPath strings pointing to the specific DOM elements included in that chunk). This metadata enables precise source attribution and downstream HTML-aware processing.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}