# Implementing Semantic Chunking for RAG: A Complete Guide to Meaning-Based Document Splitting

> Learn semantic chunking for RAG to split documents by meaning. Enhance retrieval relevance in your RAG pipelines with this essential guide.

- Repository: [NirDiamant/RAG_Techniques](https://github.com/nirdiamant/rag_techniques)
- Tags: how-to-guide
- Published: 2026-02-19

---

**Semantic chunking splits documents into meaning-based sections using embedding similarity rather than fixed character counts, significantly improving retrieval relevance in RAG pipelines.**

Implementing semantic chunking for RAG allows you to replace arbitrary text splits with intelligent boundaries that preserve context and meaning. The **NirDiamant/RAG_Techniques** repository provides a production-ready implementation through the `SemanticChunkingRAG` class, which leverages LangChain's experimental semantic chunker and FAISS vector storage to create a modular pipeline for any RAG workflow.

## What Is Semantic Chunking for RAG?

Traditional chunking methods split documents at fixed character or token boundaries, often severing sentences or semantic units mid-thought. **Semantic chunking** analyzes embedding similarities between text segments to identify natural topic boundaries, creating chunks that are self-contained and contextually coherent.

In the RAG_Techniques implementation, this process uses `langchain_experimental.text_splitter.SemanticChunker` with configurable breakpoint detection strategies (percentile, standard deviation, interquartile range, or gradient) to determine where semantic shifts occur.

## Architecture of the Semantic Chunking Implementation

The `SemanticChunkingRAG` class in [`all_rag_techniques_runnable_scripts/semantic_chunking.py`](https://github.com/NirDiamant/RAG_Techniques/blob/main/all_rag_techniques_runnable_scripts/semantic_chunking.py) orchestrates a four-stage pipeline that transforms raw PDFs into retrievable semantic chunks.

### Document Ingestion and PDF Processing

The pipeline begins with `read_pdf_to_string` from [`helper_functions.py`](https://github.com/NirDiamant/RAG_Techniques/blob/main/helper_functions.py), which uses **PyMuPDF** (`fitz`) to extract raw text from PDF files. This utility handles the initial document loading and returns a clean string representation of the entire document content.

```python
from helper_functions import read_pdf_to_string

text = read_pdf_to_string("data/Understanding_Climate_Change.pdf")

```

### Embedding Generation and Breakpoint Detection

The `SemanticChunker` initializes with an embedding model—defaulting to `OpenAIEmbeddings` but compatible with any LangChain embedding provider. The chunker analyzes embedding distances between consecutive sentences to identify semantic breakpoints based on the specified `BreakpointThresholdType`.

```python
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

embeddings = OpenAIEmbeddings()
chunker = SemanticChunker(
    embeddings,
    breakpoint_threshold_type="percentile",
    breakpoint_threshold_amount=85
)

```

### Vector Storage and Retrieval

Once chunked, documents are embedded again and stored in a **FAISS** in-memory vector index via `FAISS.from_documents`. The `as_retriever` method creates a retriever interface that returns the *k* most semantically similar chunks for any query.

```python
from langchain_community.vectorstores import FAISS

chunks = chunker.create_documents([text])
vectorstore = FAISS.from_documents(chunks, embeddings)
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})

```

## Running Semantic Chunking from the Command Line

The repository includes a full CLI interface in [`semantic_chunking.py`](https://github.com/NirDiamant/RAG_Techniques/blob/main/semantic_chunking.py) that allows immediate experimentation with different chunking parameters without writing code.

```bash
python all_rag_techniques_runnable_scripts/semantic_chunking.py \
    --path data/Understanding_Climate_Change.pdf \
    --n_retrieved 3 \
    --breakpoint_threshold_type percentile \
    --breakpoint_threshold_amount 85 \
    --query "What are the primary drivers of climate change?"

```

**CLI Arguments Explained:**

- `--path`: Path to the input PDF document.
- `--n_retrieved`: Number of semantic chunks to retrieve for the query.
- `--breakpoint_threshold_type`: Algorithm for detecting semantic shifts (`percentile`, `standard_deviation`, `interquartile`, or `gradient`).
- `--breakpoint_threshold_amount`: Sensitivity threshold (higher values produce fewer, larger chunks).
- `--query`: The natural language question used to test retrieval.

## Programmatic Implementation with Python

For integration into existing RAG pipelines, instantiate the `SemanticChunkingRAG` class directly with custom parameters.

```python
from all_rag_techniques_runnable_scripts.semantic_chunking import SemanticChunkingRAG

# Initialize with custom chunking parameters

rag = SemanticChunkingRAG(
    path="data/Understanding_Climate_Change.pdf",
    n_retrieved=2,
    breakpoint_type="percentile",
    breakpoint_amount=90,
)

# Execute retrieval and capture timing metrics

times = rag.run("How does deforestation affect global warming?")

print(f"Chunking time: {times['Chunking']:.2f}s")
print(f"Retrieval time: {times['Retrieval']:.2f}s")

```

The `run` method returns a dictionary containing execution times for the chunking and retrieval phases, enabling performance benchmarking across different breakpoint strategies.

## Key Configuration Parameters

The effectiveness of semantic chunking depends heavily on breakpoint configuration. The `SemanticChunker` supports four threshold types:

- **Percentile**: Splits when embedding distance exceeds the Nth percentile of all distances (recommended starting point: 85-95).
- **Standard Deviation**: Splits when distance exceeds the mean plus N standard deviations (useful for documents with uniform topic distribution).
- **Interquartile**: Uses the interquartile range to detect outliers in semantic distance.
- **Gradient**: Detects rapid changes in embedding similarity gradients.

According to the RAG_Techniques source code, the `percentile` method provides the most consistent results across diverse document types, while `gradient` excels with technical documents containing abrupt topic shifts.

## Summary

- **Semantic chunking** replaces fixed-size text splits with meaning-based boundaries, preserving context and improving retrieval accuracy in RAG systems.
- The **NirDiamant/RAG_Techniques** repository provides a complete implementation via the `SemanticChunkingRAG` class in [`all_rag_techniques_runnable_scripts/semantic_chunking.py`](https://github.com/NirDiamant/RAG_Techniques/blob/main/all_rag_techniques_runnable_scripts/semantic_chunking.py).
- The pipeline uses **LangChain's `SemanticChunker`** with configurable breakpoint strategies (percentile, standard deviation, interquartile, gradient) and **FAISS** for vector storage.
- Users can run the implementation via CLI arguments or integrate the `SemanticChunkingRAG` class directly into existing Python RAG pipelines.

## Frequently Asked Questions

### What is the difference between semantic chunking and recursive character chunking?

Recursive character chunking splits text at fixed character or token boundaries, often breaking sentences or ideas mid-thought. Semantic chunking analyzes embedding similarities between text segments to identify natural topic boundaries, creating chunks that are self-contained and contextually coherent. According to the RAG_Techniques implementation, semantic chunking typically yields higher retrieval relevance with fewer chunks than character-based methods.

### How do I choose the right breakpoint threshold type?

The choice depends on your document structure. Use **percentile** (85-95) for general documents with mixed topics, as implemented in the default CLI example. Choose **standard_deviation** for uniformly distributed content where outliers indicate topic shifts. Select **gradient** for technical documents with abrupt subject changes. The **interquartile** method works best when you need robust statistical outlier detection resistant to extreme values.

### Can I use different embedding models with this implementation?

Yes. The `SemanticChunkingRAG` class accepts any LangChain-compatible embedding provider. While the default implementation uses `OpenAIEmbeddings`, you can substitute Cohere, HuggingFace, Bedrock, or custom embeddings by passing the desired embedding instance to the `SemanticChunker` initialization in [`all_rag_techniques_runnable_scripts/semantic_chunking.py`](https://github.com/NirDiamant/RAG_Techniques/blob/main/all_rag_techniques_runnable_scripts/semantic_chunking.py).

### Why use FAISS as the vector store for semantic chunking?

FAISS provides fast, in-memory similarity search optimized for dense vector embeddings, making it ideal for prototyping and production RAG pipelines with semantic chunks. The RAG_Techniques implementation uses `FAISS.from_documents` to index the semantically-split chunks and `as_retriever` to configure the number of results (`k`) returned for each query, enabling sub-millisecond retrieval latency even with large document collections.