How the Text Processor Handles Chunking with Overlapping Segments for Graph Building in Mirofish

The TextProcessor class splits large documents into overlapping chunks using a sliding window algorithm that preserves sentence boundaries, ensuring no context is lost when streaming text to Zep for knowledge graph construction.

The mirofish repository implements an intelligent text chunking system designed specifically for knowledge graph generation. When processing lengthy documents, the system must balance between manageable chunk sizes for language models and the preservation of contextual relationships that span across segment boundaries. The text processor handles chunking with overlapping segments for graph building through a configurable sliding window mechanism that respects natural language breaks.

The TextProcessor API: Configurable Chunking Parameters

The public interface for document segmentation resides in backend/app/services/text_processor.py. The TextProcessor.split_text method serves as the primary entry point, accepting two tunable parameters that control how the text processor handles chunking with overlapping segments for graph building.

split_text Method Signature and Defaults

The method signature defines sensible defaults while allowing customization for different document types:

# backend/app/services/text_processor.py

def split_text(text: str, chunk_size: int = 500, overlap: int = 50) -> List[str]:
    """
    Split text into chunks with specified size and overlap.
    
    Args:
        text: The input text to split
        chunk_size: Target number of characters per chunk (default 500)
        overlap: Number of characters to overlap between chunks (default 50)
    
    Returns:
        List of text chunks with overlapping segments
    """
    return split_text_into_chunks(text, chunk_size, overlap)

Source: text_processor.py#L18-L34

The default chunk_size of 500 characters provides a balance between context richness and processing efficiency, while the default overlap of 50 characters ensures that entities or relationships mentioned near chunk boundaries appear in both adjacent segments.

The Sliding Window Algorithm with Sentence Boundary Detection

The actual chunking implementation resides in backend/app/utils/file_parser.py within the split_text_into_chunks function. This algorithm demonstrates how the text processor handles chunking with overlapping segments for graph building through a sophisticated sliding window that respects linguistic boundaries.

How Overlapping Segments Preserve Context

The core mechanism uses a while-loop that advances through the document with overlapping windows:

# backend/app/utils/file_parser.py

def split_text_into_chunks(text: str, chunk_size: int, overlap: int) -> List[str]:
    chunks = []
    start = 0
    text_length = len(text)
    
    while start < text_length:
        # Calculate initial end position

        end = start + chunk_size
        
        # If we're not at the end of text, try to find a natural break

        if end < text_length:
            # Look for sentence-ending punctuation after 30% of chunk_size

            search_start = start + int(chunk_size * 0.3)
            search_end = min(end + 50, text_length)  # Look a bit ahead

            substring = text[search_start:search_end]
            
            # Find the last sentence-ending punctuation

            punctuations = ['。', '！', '？', '.', '!', '?']
            last_punct_pos = -1
            for punct in punctuations:
                pos = substring.rfind(punct)
                if pos > last_punct_pos:
                    last_punct_pos = pos
            
            if last_punct_pos != -1:
                end = search_start + last_punct_pos + 1
        
        # Extract and clean the chunk

        chunk = text[start:end].strip()
        if chunk:
            chunks.append(chunk)
        
        # Move start position for next chunk, accounting for overlap

        start = end - overlap if end < text_length else text_length
    
    return chunks

Source: file_parser.py#L47-L88

The critical line start = end - overlap creates the overlapping effect. By subtracting the overlap value from the previous end position, the next chunk begins overlap characters before the previous chunk ended, ensuring that any entities or relationships mentioned near the boundary appear in both chunks.

Natural Language Breaks at Punctuation

The algorithm implements sentence boundary detection to avoid cutting chunks mid-sentence. When the window does not reach the end of the document, the code searches for sentence-ending punctuation marks (。!？.!?) within a search window that starts at 30% of the chunk size and extends slightly beyond the target end.

This approach ensures that:

• Chunks maintain semantic coherence by ending at natural boundaries
• The overlap mechanism captures complete sentences when possible
• Graph extraction receives linguistically meaningful segments rather than arbitrary character cuts

Integration with Graph Building Pipeline

The overlapping chunks generated by the text processor flow directly into the knowledge graph construction workflow. Understanding how the text processor handles chunking with overlapping segments for graph building requires examining the consumption point in backend/app/services/graph_builder.py.

From Raw Text to Zep Streaming

The GraphBuilderService orchestrates the pipeline by invoking the text processor and streaming results to Zep:

# backend/app/services/graph_builder.py

class GraphBuilderService:
    def __init__(self, api_key: str):
        self.api_key = api_key
        self.zep_client = ZepClient(api_key)
    
    def build_graph_async(
        self,
        text: str,
        ontology: Dict,
        graph_name: str,
        chunk_size: int = 500,
        chunk_overlap: int = 50,
        batch_size: int = 3
    ) -> str:
        """
        Asynchronously build a knowledge graph from text.
        
        Args:
            text: Raw document text
            ontology: Entity and relationship schema
            graph_name: Identifier for the graph
            chunk_size: Characters per chunk (passed to TextProcessor)
            chunk_overlap: Overlap between chunks (passed to TextProcessor)
            batch_size: Number of chunks to send per batch to Zep
        
        Returns:
            Task ID for tracking progress
        """
        # Split text using the overlapping chunk processor

        chunks = TextProcessor.split_text(text, chunk_size, chunk_overlap)
        
        # Stream chunks to Zep for entity extraction

        task_id = self.zep_client.add_text_batches(
            graph_name=graph_name,
            ontology=ontology,
            text_chunks=chunks,
            batch_size=batch_size
        )
        
        return task_id

Source: graph_builder.py#L30-L33

The overlapping segments ensure that entity relationships spanning chunk boundaries are captured by the language model. When Zep processes these chunks, the duplicated content at the overlap regions provides sufficient context for the model to recognize connections that might otherwise be split across separate processing batches.

Practical Implementation Examples

Direct Text Chunking with Custom Parameters

You can leverage the text processor directly for document preprocessing:

from backend.app.services.text_processor import TextProcessor

# Sample Chinese historical text

sample = (
    "在古代，中国的四大发明对世界产生了深远影响。"
    "造纸术、印刷术、指南针和火药分别推动了信息传播、"
    "技术革新、航海探索和武器发展。"
)

# Create smaller chunks with substantial overlap for dense entity extraction

chunks = TextProcessor.split_text(sample, chunk_size=30, overlap=10)

for i, chunk in enumerate(chunks, 1):
    print(f"Chunk {i}: '{chunk}'")

Output characteristics:

• Chunk 1 ends with the last 10 characters appearing at the start of Chunk 2
• Sentence boundaries (。) are respected when falling within the search window
• Each chunk maintains semantic coherence despite the aggressive size constraints

Integrating with GraphBuilderService

For production graph construction, configure the chunking parameters based on your ontology complexity:

from backend.app.services.graph_builder import GraphBuilderService

# Initialize the service with your Zep API credentials

service = GraphBuilderService(api_key="your_zep_api_key")

# Process a complex legal document requiring high context preservation

task_id = service.build_graph_async(
    text=legal_document_text,
    ontology=legal_ontology,
    graph_name="Legal Entities Graph",
    chunk_size=400,      # Smaller chunks for precise entity location

    chunk_overlap=100,   # Larger overlap to preserve cross-boundary relationships

    batch_size=2         # Conservative batching for API rate limits

)

print(f"Graph construction initiated: {task_id}")

The service internally invokes TextProcessor.split_text with your specified chunk_size and chunk_overlap values, then streams the resulting overlapping segments to Zep's entity extraction pipeline.

Summary

The mirofish text processor handles chunking with overlapping segments for graph building through a sophisticated sliding window algorithm that balances mechanical efficiency with linguistic coherence:

• Configurable parameters via TextProcessor.split_text in backend/app/services/text_processor.py allow tuning of chunk_size (default 500) and overlap (default 50) to match document characteristics.
• Sentence-aware boundaries in split_text_into_chunks (backend/app/utils/file_parser.py) search for punctuation marks (。!？.!?) to ensure chunks end at natural linguistic breaks rather than arbitrary character counts.
• Context preservation through the overlap mechanism ensures that the final overlap characters of each chunk reappear at the start of the next, preventing entity relationships that span chunk boundaries from being lost during graph construction.
• Pipeline integration via GraphBuilderService (backend/app/services/graph_builder.py) streams these overlapping segments to Zep for entity extraction, leveraging the duplicated context to improve relationship detection across adjacent chunks.

Frequently Asked Questions

What is the default chunk size and overlap in Mirofish?

The default configuration in TextProcessor.split_text uses a chunk_size of 500 characters and an overlap of 50 characters. These defaults strike a balance between providing sufficient context for entity extraction (500 characters typically covers multiple sentences) and maintaining processing efficiency, while the 50-character overlap ensures that words or relationships near chunk boundaries appear in both adjacent segments.

How does the algorithm prevent cutting words in the middle?

The chunking algorithm implements sentence boundary detection within split_text_into_chunks (backend/app/utils/file_parser.py). When a chunk window does not reach the end of the document, the code searches for sentence-ending punctuation marks—including Chinese periods (。), exclamation points (！), question marks (？), and their English equivalents—within a search window starting at 30% of the target chunk size. If found, the chunk end is adjusted to that punctuation mark, ensuring chunks end at natural linguistic boundaries rather than mid-word.

Why is overlapping important for knowledge graph construction?

Overlapping segments are critical for preserving cross-boundary relationships during entity extraction. When GraphBuilderService streams chunks to Zep for processing, language models analyze each chunk independently to extract entities and relationships. Without overlap, an entity mentioned at the very end of chunk n and referenced at the start of chunk n+1 would appear in separate, disconnected contexts. The overlap ensures these boundary-spanning references appear in both chunks, allowing the graph construction process to maintain relationship continuity across the entire document.

Can I adjust chunking parameters for different document types?

Yes, the chunking parameters are fully configurable when invoking either TextProcessor.split_text directly or through GraphBuilderService.build_graph_async. For dense technical documents requiring high precision, you might reduce chunk_size to 300-400 characters and increase overlap to 100 characters to ensure technical terms remain contextualized. For narrative text with longer paragraphs, you might increase chunk_size to 800-1000 characters while maintaining a proportional overlap of 10% (80-100 characters). These adjustments are passed through the chunk_size and chunk_overlap parameters in the graph building pipeline.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}