# How to Combine betterhtmlchunking with LangChain for Hybrid RAG Pipelines
> Integrate betterhtmlchunking and LangChain for hybrid RAG pipelines. Learn to create structure-aware chunks from HTML ARoIs for advanced retrieval-augmented generation workflows.
- Repository: [Carlos A. Planchón/betterhtmlchunking](https://github.com/carlosplanchon/betterhtmlchunking)
- Tags: how-to-guide
- Published: 2026-02-26
---
**You can combine betterhtmlchunking with LangChain by wrapping each Region of Interest (ROI) from the HTML parser into a LangChain `Document` object, enabling structure-aware chunking with XPath provenance for hybrid retrieval-augmented generation workflows.**
The `carlosplanchon/betterhtmlchunking` library parses HTML documents into semantically coherent chunks while preserving structural metadata. When building hybrid RAG pipelines with LangChain, you need to bridge these DOM-aware chunks with LangChain's expected `Document` format to enable vector storage, retrieval, and downstream LLM consumption.
## Understanding the betterhtmlchunking Architecture
The library processes HTML through a three-stage pipeline defined across several core modules. Understanding these components helps you correctly extract and format chunks for LangChain integration.
### DOM Representation and ROI Detection
The `DomRepresentation` class in [`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py) serves as the primary facade. It orchestrates the parsing of HTML into a DOM tree and computes **Regions of Interest (ROIs)**—groupings of nodes that respect your defined size limits while maintaining structural boundaries.
```python
from betterhtmlchunking import DomRepresentation, ReprLengthComparisionBy
dom_repr = DomRepresentation(
MAX_NODE_REPR_LENGTH=2000, # size limit per ROI
website_code=html_source,
repr_length_compared_by=ReprLengthComparisionBy.HTML_LENGTH
)
dom_repr.start()
```
The `DomRepresentation.start()` method triggers the pipeline stages defined in [`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py), where the `TreeRegionsSystem` and `ROIMaker` classes implement the core algorithm for grouping nodes into size-constrained regions.
### Rendering System
Once ROIs are identified, the `RenderSystem` class in [`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py) produces two parallel dictionaries:
- `html_render_roi[idx]` – The full HTML representation of the region
- `text_render_roi[idx]` – The plain-text extraction of the same region
These dictionaries map region indices to their respective content formats, enabling you to provide both structured HTML and clean text to LangChain.
## Integrating with LangChain Documents
LangChain's RAG workflows expect objects containing `page_content` (text for the LLM) and `metadata` (arbitrary key-value pairs for provenance). You bridge betterhtmlchunking to LangChain by mapping each ROI to a `Document` instance.
### Conversion Function
The following function converts the output of `DomRepresentation` into a list of LangChain `Document` objects:
```python
from betterhtmlchunking import DomRepresentation, ReprLengthComparisionBy
from langchain.docstore.document import Document
def html_to_langchain_docs(
html_source: str,
max_len: int = 2000,
compare_by: ReprLengthComparisionBy = ReprLengthComparisionBy.HTML_LENGTH,
) -> list[Document]:
"""Convert raw HTML into LangChain Document objects using betterhtmlchunking."""
# Build DOM representation and run the pipeline
dom = DomRepresentation(
MAX_NODE_REPR_LENGTH=max_len,
website_code=html_source,
repr_length_compared_by=compare_by,
)
dom.start()
# Pull rendered chunks
html_chunks = dom.render_system.html_render_roi
text_chunks = dom.render_system.text_render_roi
rois = dom.tree_regions_system.sorted_roi_by_pos_xpath
# Wrap each ROI in a LangChain Document
docs: list[Document] = []
for idx, _ in html_chunks.items():
docs.append(
Document(
page_content=text_chunks[idx],
metadata={
"chunk_index": idx,
"html": html_chunks[idx],
"xpaths": rois[idx].pos_xpath_list,
},
)
)
return docs
```
This implementation preserves the XPath provenance in `metadata["xpaths"]`, stores the original HTML in `metadata["html"]`, and uses the plain-text version as `page_content` for embedding and LLM consumption.
## Building the Hybrid RAG Pipeline
With the conversion function in place, you can construct a complete hybrid RAG pipeline that leverages structure-aware chunking for retrieval and optional token-level splitting for generation.
### Vector Store Indexing
Index the documents using your preferred vector store and embedding model:
```python
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import FAISS
# Load and chunk HTML
with open("documentation.html", "r", encoding="utf-8") as f:
html_raw = f.read()
docs = html_to_langchain_docs(html_raw, max_len=1500)
# Create vector store
embeddings = OpenAIEmbeddings()
vectorstore = FAISS.from_documents(docs, embedding=embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
```
### Retrieval and Optional Token Splitting
Retrieve relevant chunks and optionally apply secondary token-based splitting if the structural chunks exceed your LLM's context window:
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
query = "What are the deployment requirements for the API?"
relevant_docs = retriever.get_relevant_documents(query)
# Hybrid approach: split only if necessary
splitter = RecursiveCharacterTextSplitter(chunk_size=1024, chunk_overlap=200)
final_chunks = []
for doc in relevant_docs:
if len(doc.page_content) > 1024:
final_chunks.extend(splitter.split_documents([doc]))
else:
final_chunks.append(doc)
```
### End-to-End Integration Example
Combine all components into a complete retrieval chain:
```python
from langchain.llms import OpenAI
from langchain.chains import RetrievalQA
# Initialize LLM and QA chain
llm = OpenAI()
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
retriever=retriever,
return_source_documents=True,
)
# Execute query
result = qa_chain.run("Explain the authentication flow")
print(result["result"])
# Access provenance
for source in result["source_documents"]:
print(f"Chunk {source.metadata['chunk_index']}: {source.metadata['xpaths']}")
```
## Key Source Files and Implementation Details
The following files in the `carlosplanchon/betterhtmlchunking` repository implement the core functionality referenced above:
- **[`betterhtmlchunking/main.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/main.py)** – Contains the `DomRepresentation` class that serves as the primary API entry point, orchestrating the pipeline through its `start()` method and `__attrs_post_init__` initialization.
- **[`betterhtmlchunking/tree_representation.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_representation.py)** – Implements the DOM tree construction, node length computation (both plain-text and HTML), and XPath-to-node mapping.
- **[`betterhtmlchunking/tree_regions_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/tree_regions_system.py)** – Houses the `TreeRegionsSystem` and `ROIMaker` classes that implement the region detection algorithm, grouping nodes into size-constrained ROIs while preserving structural boundaries.
- **[`betterhtmlchunking/render_system.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/render_system.py)** – Provides the `RenderSystem` class that generates parallel HTML and plain-text dictionaries for each ROI index.
- **[`betterhtmlchunking/utils.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/utils.py)** – Contains helper functions for filtering unwanted tags before chunking.
- **[`betterhtmlchunking/__init__.py`](https://github.com/carlosplanchon/betterhtmlchunking/blob/main/betterhtmlchunking/__init__.py)** – Exports the public API including `DomRepresentation` and `ReprLengthComparisionBy`.
## Summary
- **betterhtmlchunking** parses HTML into structure-aware **Regions of Interest (ROIs)** that respect size limits while preserving DOM boundaries and XPath provenance.
- You can integrate these ROIs with LangChain by wrapping each region in a `Document` object, using the plain-text content for `page_content` and storing HTML snippets, XPaths, and indices in `metadata`.
- This approach enables **hybrid RAG pipelines** that combine structural chunking for semantic retrieval with optional token-based splitting for generation, ensuring precise provenance traceability back to specific HTML elements.
## Frequently Asked Questions
### What is the primary advantage of using betterhtmlchunking over standard text splitters?
Standard text splitters like LangChain's `RecursiveCharacterTextSplitter` treat HTML as raw text, often splitting across semantic boundaries like paragraphs or list items. **betterhtmlchunking** respects the DOM structure, ensuring that related elements stay together in the same chunk and preserving XPath provenance for traceability back to the source HTML.
### How does the hybrid approach handle oversized HTML regions?
If a structural ROI from betterhtmlchunking exceeds your LLM's token limit, you can apply a secondary **token-based splitter** (such as `RecursiveCharacterTextSplitter`) only to those specific chunks. This "structure first, token-size second" approach maintains semantic coherence while ensuring content fits within context windows.
### Can I use betterhtmlchunking with other LLM frameworks besides LangChain?
Yes. The library outputs standard Python dictionaries containing HTML snippets, plain text, and XPath lists. You can adapt the conversion logic to create document objects for **LlamaIndex**, **Haystack**, or custom frameworks by mapping the ROI fields to the respective framework's document schema.
### What metadata is available for each chunk in the LangChain Document?
Each `Document` object contains a `metadata` dictionary with three key fields: **`chunk_index`** (the integer ID of the ROI), **`html`** (the full HTML snippet of the region), and **`xpaths`** (a list of XPath strings pointing to the specific DOM elements included in that chunk). This metadata enables precise source attribution and downstream HTML-aware processing.