How the TextChunker Implements Section-Based Chunking with Overlap

The TextChunker class uses a hybrid strategy that preserves natural document boundaries by chunking according to paper sections, then applies configurable word-level overlap only when individual sections exceed size thresholds.

The TextChunker service, located in src/services/indexing/text_chunker.py within the jamwithai/production-agentic-rag-course repository, provides a production-ready implementation of semantic text splitting designed specifically for academic papers. Unlike naive character-splitting approaches, this chunker first attempts to respect the paper's original section structure, only falling back to overlapping word-based segmentation when sections grow too large for vector embedding contexts.

The Hybrid Section-Based Architecture

The entry point chunk_paper orchestrates a multi-stage pipeline that prioritizes semantic coherence over arbitrary text splits. When you invoke chunk_paper(title, abstract, full_text, sections), the method first attempts section-based processing through _chunk_by_sections. If the sections payload is malformed or empty, it automatically falls back to generic word-based chunking to ensure robustness.

This architecture recognizes that scientific papers contain natural semantic boundaries—titles, abstracts, and section headers—that should remain intact when possible. The implementation classifies each section by word count to determine whether it can stand alone as a single chunk or requires subdivision.

Parsing and Filtering Section Metadata

Before chunking begins, the _chunk_by_sections method delegates preprocessing to two private helpers. First, _parse_sections normalizes the input sections payload, handling dict, JSON string, or list formats flexibly. Then, _filter_sections removes metadata cruft and eliminates duplicate abstract content that might appear in both the dedicated abstract field and the full-text sections.

This cleaning step ensures that when the chunker builds its contextual header (combining the paper title and abstract), it does not inadvertently duplicate information already present in the body sections. The filtered sections are then evaluated individually in the main processing loop.

Size-Based Classification Strategy

The core classification logic inside _chunk_by_sections categorizes each section into one of three size classes:

• Small sections (< 100 words): Buffered and later merged via _create_combined_chunk to avoid creating tiny, low-information chunks
• Medium sections (100–800 words): Converted directly into single chunks via _create_section_chunk, preserving the complete section context
• Large sections (> 800 words): Delegated to _split_large_section for overlapping subdivision

This threshold-based approach ensures that appropriately-sized sections remain unsplit, maintaining semantic coherence, while only oversized content gets fragmented. The word count thresholds are configurable through the class initialization, though the defaults target typical embedding model context windows.

Implementing Sliding-Window Overlap

When large sections require splitting, the implementation relies on the chunk_text method—the generic workhorse for all overlapping splits. This method implements a standard sliding window algorithm that guarantees continuity between adjacent chunks.

The chunk_text method first tokenizes input into words via _split_into_words, then iterates while current_position < len(words). For each iteration, it extracts a slice of chunk_size words, calculates precise character offsets, and records overlap_with_previous and overlap_with_next values based on self.overlap_size. Crucially, the cursor advances by chunk_size - overlap_size, ensuring that consecutive chunks share exactly the configured number of words. This overlap prevents semantic discontinuities at chunk boundaries, which is critical for retrieval-augmented generation (RAG) applications.

Handling Large Sections with Context Preservation

The _split_large_section method manages the transition from section-aware to word-based processing. Before calling chunk_text, it strips the common header (title and abstract) from the section body to prevent duplication. After splitting, it re-attaches this header to each resulting chunk and updates the metadata to include the original section_title with a "(Part n)" suffix.

This technique ensures that every fragment of a large section carries the document-level context necessary for accurate retrieval, while the metadata enrichment tracks the chunk's position within the original section. Each chunk receives a ChunkMetadata record containing index, character offsets, word counts, and overlap values defined in src/schemas/indexing/models.py.

Complete Implementation Example

The following example demonstrates initializing the chunker with custom overlap settings and processing a paper with mixed section sizes:

from src.services.indexing.text_chunker import TextChunker

# Initialize with 600-word chunks and 100-word overlap

chunker = TextChunker(chunk_size=600, overlap_size=100)

# Sample paper data

title = "A Study of AI Alignment"
abstract = "We investigate methods for aligning powerful AI systems with human values."
full_text = """...very long body of the paper..."""
sections = {
    "Introduction": "The field of AI alignment ...",
    "Related Work": "Previous research ...",
    "Methodology": "We propose a novel framework ... " * 500,  # >800 words

    "Conclusion": "Our experiments show ..."
}

# Generate chunks respecting sections where possible

chunks = chunker.chunk_paper(
    title=title,
    abstract=abstract,
    full_text=full_text,
    arxiv_id="2101.12345",
    paper_id="paper-001",
    sections=sections,
)

# Inspect results

for i, ch in enumerate(chunks[:3]):
    print(f"Chunk {i} – Section: {ch.metadata.section_title}")
    print(f"Words: {ch.metadata.word_count}, OverlapPrev: {ch.metadata.overlap_with_previous}")
    print(ch.text[:200], "...\n")

In this example, the medium sections (Introduction, Related Work, Conclusion) each become single chunks through _create_section_chunk, while the large Methodology section triggers _split_large_section, producing multiple overlapping chunks with the "(Part n)" designation in their metadata.

Summary

• The TextChunker attempts section-based chunking first via _chunk_by_sections, falling back to generic word-based splitting only when necessary
• Sections are classified by word count: small sections (<100 words) are merged, medium sections (100-800 words) are preserved whole, and large sections (>800 words) are split
• Overlap is implemented through a sliding window in chunk_text, advancing by chunk_size - overlap_size to ensure configurable word-level continuity between chunks
• Large sections have their headers temporarily removed before splitting, then re-attached to each fragment to maintain document context
• All chunks carry enriched ChunkMetadata including character offsets, word counts, and overlap statistics

Frequently Asked Questions

How does the chunker handle sections smaller than 100 words?

Sections with fewer than 100 words are classified as small and buffered rather than immediately chunked. The _create_combined_chunk method merges these small sections with adjacent content, preventing the creation of tiny, semantically impoverished chunks that could degrade retrieval quality.

What metadata is preserved in each chunk?

Every chunk receives a ChunkMetadata object containing the chunk index, start and end character offsets, word count, overlap_with_previous and overlap_with_next values, and the originating section_title. For split sections, the title includes the "(Part n)" suffix to indicate position within the original section.

How is the overlap size configured and applied?

The overlap size is set during TextChunker initialization via the overlap_size parameter (defaulting to 100 words). This value controls how many words overlap between consecutive chunks in the chunk_text sliding window. The cursor advances by chunk_size - overlap_size, meaning a 600-word chunk with 100-word overlap advances 500 words, sharing exactly 100 words with the previous chunk.

Why does the implementation remove and re-add headers when splitting large sections?

The _split_large_section method strips the common document header (title and abstract) before calling chunk_text to prevent redundant context from appearing in every sub-chunk. After splitting, it re-attaches this header to each fragment. This ensures that while each chunk maintains document-level context for retrieval accuracy, the splitting algorithm operates only on the actual section content, preventing the header from consuming valuable tokens in the word-count calculations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}