# How the TextChunker Implements Section-Based Chunking with Overlap

> Discover how the TextChunker implements section-based chunking with overlap. Learn to preserve document boundaries and optimize text segmentation for RAG.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: internals
- Published: 2026-03-23

---

**The `TextChunker` class uses a hybrid strategy that preserves natural document boundaries by chunking according to paper sections, then applies configurable word-level overlap only when individual sections exceed size thresholds.**

The `TextChunker` service, located in [`src/services/indexing/text_chunker.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/indexing/text_chunker.py) within the `jamwithai/production-agentic-rag-course` repository, provides a production-ready implementation of semantic text splitting designed specifically for academic papers. Unlike naive character-splitting approaches, this chunker first attempts to respect the paper's original section structure, only falling back to overlapping word-based segmentation when sections grow too large for vector embedding contexts.

## The Hybrid Section-Based Architecture

The entry point `chunk_paper` orchestrates a multi-stage pipeline that prioritizes semantic coherence over arbitrary text splits. When you invoke `chunk_paper(title, abstract, full_text, sections)`, the method first attempts section-based processing through `_chunk_by_sections`. If the sections payload is malformed or empty, it automatically falls back to generic word-based chunking to ensure robustness.

This architecture recognizes that scientific papers contain natural semantic boundaries—titles, abstracts, and section headers—that should remain intact when possible. The implementation classifies each section by word count to determine whether it can stand alone as a single chunk or requires subdivision.

## Parsing and Filtering Section Metadata

Before chunking begins, the `_chunk_by_sections` method delegates preprocessing to two private helpers. First, `_parse_sections` normalizes the input sections payload, handling dict, JSON string, or list formats flexibly. Then, `_filter_sections` removes metadata cruft and eliminates duplicate abstract content that might appear in both the dedicated abstract field and the full-text sections.

This cleaning step ensures that when the chunker builds its contextual header (combining the paper title and abstract), it does not inadvertently duplicate information already present in the body sections. The filtered sections are then evaluated individually in the main processing loop.

## Size-Based Classification Strategy

The core classification logic inside `_chunk_by_sections` categorizes each section into one of three size classes:

- **Small sections** (< 100 words): Buffered and later merged via `_create_combined_chunk` to avoid creating tiny, low-information chunks
- **Medium sections** (100–800 words): Converted directly into single chunks via `_create_section_chunk`, preserving the complete section context
- **Large sections** (> 800 words): Delegated to `_split_large_section` for overlapping subdivision

This threshold-based approach ensures that appropriately-sized sections remain unsplit, maintaining semantic coherence, while only oversized content gets fragmented. The word count thresholds are configurable through the class initialization, though the defaults target typical embedding model context windows.

## Implementing Sliding-Window Overlap

When large sections require splitting, the implementation relies on the `chunk_text` method—the generic workhorse for all overlapping splits. This method implements a standard sliding window algorithm that guarantees continuity between adjacent chunks.

The `chunk_text` method first tokenizes input into words via `_split_into_words`, then iterates while `current_position < len(words)`. For each iteration, it extracts a slice of `chunk_size` words, calculates precise character offsets, and records `overlap_with_previous` and `overlap_with_next` values based on `self.overlap_size`. Crucially, the cursor advances by `chunk_size - overlap_size`, ensuring that consecutive chunks share exactly the configured number of words. This overlap prevents semantic discontinuities at chunk boundaries, which is critical for retrieval-augmented generation (RAG) applications.

## Handling Large Sections with Context Preservation

The `_split_large_section` method manages the transition from section-aware to word-based processing. Before calling `chunk_text`, it strips the common header (title and abstract) from the section body to prevent duplication. After splitting, it re-attaches this header to each resulting chunk and updates the metadata to include the original `section_title` with a "(Part n)" suffix.

This technique ensures that every fragment of a large section carries the document-level context necessary for accurate retrieval, while the metadata enrichment tracks the chunk's position within the original section. Each chunk receives a `ChunkMetadata` record containing index, character offsets, word counts, and overlap values defined in [`src/schemas/indexing/models.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/schemas/indexing/models.py).

## Complete Implementation Example

The following example demonstrates initializing the chunker with custom overlap settings and processing a paper with mixed section sizes:

```python
from src.services.indexing.text_chunker import TextChunker

# Initialize with 600-word chunks and 100-word overlap

chunker = TextChunker(chunk_size=600, overlap_size=100)

# Sample paper data

title = "A Study of AI Alignment"
abstract = "We investigate methods for aligning powerful AI systems with human values."
full_text = """...very long body of the paper..."""
sections = {
    "Introduction": "The field of AI alignment ...",
    "Related Work": "Previous research ...",
    "Methodology": "We propose a novel framework ... " * 500,  # >800 words

    "Conclusion": "Our experiments show ..."
}

# Generate chunks respecting sections where possible

chunks = chunker.chunk_paper(
    title=title,
    abstract=abstract,
    full_text=full_text,
    arxiv_id="2101.12345",
    paper_id="paper-001",
    sections=sections,
)

# Inspect results

for i, ch in enumerate(chunks[:3]):
    print(f"Chunk {i} – Section: {ch.metadata.section_title}")
    print(f"Words: {ch.metadata.word_count}, OverlapPrev: {ch.metadata.overlap_with_previous}")
    print(ch.text[:200], "...\n")

```

In this example, the **medium** sections (`Introduction`, `Related Work`, `Conclusion`) each become single chunks through `_create_section_chunk`, while the **large** `Methodology` section triggers `_split_large_section`, producing multiple overlapping chunks with the `"(Part n)"` designation in their metadata.

## Summary

- The `TextChunker` attempts section-based chunking first via `_chunk_by_sections`, falling back to generic word-based splitting only when necessary
- Sections are classified by word count: small sections (<100 words) are merged, medium sections (100-800 words) are preserved whole, and large sections (>800 words) are split
- Overlap is implemented through a sliding window in `chunk_text`, advancing by `chunk_size - overlap_size` to ensure configurable word-level continuity between chunks
- Large sections have their headers temporarily removed before splitting, then re-attached to each fragment to maintain document context
- All chunks carry enriched `ChunkMetadata` including character offsets, word counts, and overlap statistics

## Frequently Asked Questions

### How does the chunker handle sections smaller than 100 words?

Sections with fewer than 100 words are classified as **small** and buffered rather than immediately chunked. The `_create_combined_chunk` method merges these small sections with adjacent content, preventing the creation of tiny, semantically impoverished chunks that could degrade retrieval quality.

### What metadata is preserved in each chunk?

Every chunk receives a `ChunkMetadata` object containing the chunk index, start and end character offsets, word count, `overlap_with_previous` and `overlap_with_next` values, and the originating `section_title`. For split sections, the title includes the "(Part n)" suffix to indicate position within the original section.

### How is the overlap size configured and applied?

The overlap size is set during `TextChunker` initialization via the `overlap_size` parameter (defaulting to 100 words). This value controls how many words overlap between consecutive chunks in the `chunk_text` sliding window. The cursor advances by `chunk_size - overlap_size`, meaning a 600-word chunk with 100-word overlap advances 500 words, sharing exactly 100 words with the previous chunk.

### Why does the implementation remove and re-add headers when splitting large sections?

The `_split_large_section` method strips the common document header (title and abstract) before calling `chunk_text` to prevent redundant context from appearing in every sub-chunk. After splitting, it re-attaches this header to each fragment. This ensures that while each chunk maintains document-level context for retrieval accuracy, the splitting algorithm operates only on the actual section content, preventing the header from consuming valuable tokens in the word-count calculations.