How Semantica's EntityAwareChunker Preserves Entity Boundaries: A Complete Implementation Guide

Semantica's EntityAwareChunker preserves entity boundaries by extracting named entities first, building a boundary map of their character offsets, then deferring chunk splits whenever a sentence intersects an entity span.

The EntityAwareChunker in the semantica-agi/semantica repository solves a critical problem in document chunking for GraphRAG and knowledge-graph pipelines: preventing named entities from being split across chunks. This article explains how the implementation guarantees semantic coherence by keeping entity spans intact.

Core Architecture of Entity Boundary Preservation

The chunker operates through a multi-stage pipeline defined across two primary files. Understanding this architecture helps you configure and extend the behavior for your own NER pipelines.

Initialization and Configuration

In semantica/split/kg_chunkers.py (lines 62-78), the EntityAwareChunker class stores the operational parameters that control boundary preservation:

# Key parameters from kg_chunkers.py

chunk_size: int        # Target character length per chunk

chunk_overlap: int      # Overlap between consecutive chunks

ner_method: str         # "ml", "spacy", or other NER backend

preserve_entities: bool # Master flag for boundary protection

The preserve_entities flag serves as the master switch—when False, the chunker reverts to standard sentence-wise splitting without boundary checks.

Entity Extraction and Boundary Mapping

The critical boundary-preservation logic lives in semantica/split/methods.py. Here's how the algorithm builds its protective map:

Step 1: Extract entity spans

# From methods.py lines 78-82

ner_extractor = NERExtractor(method=self.ner_method)
entities = ner_extractor.extract(text)  # List of objects with start_char, end_char

Step 2: Build the boundary set

# From methods.py lines 84-88

entity_boundaries = set()
for entity in entities:
    entity_boundaries.add(entity.start_char)
    entity_boundaries.add(entity.end_char)

This set structure enables O(1) lookup when testing whether any point in the text falls inside an entity boundary.

The Boundary-Aware Splitting Algorithm

The core innovation is the deferred split mechanism that triggers during sentence-wise iteration.

Sentence Processing Loop

From methods.py lines 95-118, the algorithm:

1. Splits text into sentences using _split_sentences_regex
2. Tracks character ranges for each sentence (sentence_start, sentence_end)
3. Tests for boundary intersection:

# From methods.py lines 103-107

has_entity_boundary = any(
    sentence_start < boundary <= sentence_end
    for boundary in entity_boundaries
)

4. Conditionally defers the split:

# From methods.py lines 110-118 — the critical decision block

if current_length + sentence_length > self.chunk_size:
    if self.preserve_entities and has_entity_boundary:
        # DEFER: Don't split here; keep entity whole

        current_chunk.append(sentence)
    else:
        # SPLIT: Safe to break at this sentence boundary

        emit_chunk(current_chunk)
        current_chunk = [sentence]

This conditional logic is the engine of entity preservation. Even when the target chunk_size would be exceeded, the algorithm extends the current chunk rather than bisecting an entity.

Final Chunk Emission

The algorithm handles trailing text in methods.py lines 155-165, ensuring the last chunk also respects entity boundaries.

Chunk Metadata and Downstream Integration

Emitted chunks carry rich metadata that downstream GraphRAG components consume. From methods.py lines 124-133:

chunk = Chunk(
    text="".join(current_chunk_sentences),
    start_index=chunk_start,
    end_index=chunk_end,
    metadata={
        "method": "entity_aware",
        "entity_count": len(entities_in_chunk),
        "entities": entities_in_chunk  # Full entity objects with offsets

    }
)

This metadata enables precise entity-to-chunk mapping without re-running NER.

Complete Working Example

from semantica.split.kg_chunkers import EntityAwareChunker

text = """
Apple Inc. announced its quarterly earnings on Tuesday. The company reported 
revenue of $90.1 billion for Q4 2023, exceeding analyst expectations. 
Tim Cook, CEO of Apple Inc., highlighted strong iPhone 15 sales across 
all markets. The board declared a dividend of $0.24 per share.
"""

# Initialize with ML-based NER and strict boundary preservation

chunker = EntityAwareChunker(
    chunk_size=120,
    chunk_overlap=20,
    ner_method="ml",
    preserve_entities=True  # Explicitly enable boundary protection

)

chunks = chunker.chunk(text)

for i, ch in enumerate(chunks, 1):
    print(f"\n--- Chunk {i} ({ch.start_index}-{ch.end_index}) ---")
    print(ch.text[:200] + "...")
    print("Preserved entities:", [e.text for e in ch.metadata["entities"]])

Output guarantee: Multi-word entities like "Apple Inc." and "Tim Cook" never appear fragmented across chunks, even when strict size limits would otherwise force a mid-entity split.

Key Implementation Files

File	Purpose	Critical Lines
semantica/split/kg_chunkers.py	EntityAwareChunker class definition	62-78 (initialization)
semantica/split/methods.py	split_entity_aware() core algorithm	78-133, 155-165
semantica/split/semantic_chunker.py	Chunk dataclass with metadata	Full file
semantica/utils/logging.py	Diagnostic logger	get_logger usage
semantica/utils/progress_tracker.py	Operation lifecycle tracking	Integration points

Performance Characteristics

• Boundary testing: O(1) per sentence due to set lookup
• Overall complexity: O(n) where n = document character length
• Memory overhead: Proportional to entity count (boundary set) plus sentence list
• Size guarantee: Chunks may exceed chunk_size when entities are longer than the target; no hard truncation occurs

Summary

• EntityAwareChunker preserves boundaries by extracting entities first, then deferring splits that would intersect entity spans
• The boundary map (entity_boundaries set) enables fast intersection tests during sentence-wise iteration
• The deferred split mechanism in methods.py lines 110-118 is the core protective logic
• Chunks carry rich metadata including full entity objects for downstream GraphRAG pipelines
• The preserve_entities flag allows runtime toggle between strict and standard chunking

Frequently Asked Questions

What happens if an entity is longer than the chunk_size?

The algorithm emits the entity in a single chunk that exceeds the target size. Boundary preservation takes priority over size constraints—no hard truncation occurs. This prevents semantic fragmentation at the cost of occasional oversized chunks.

Can I use a custom NER model with EntityAwareChunker?

Yes. The ner_method parameter accepts any string registered with the NERExtractor factory. Implement a custom extractor following the interface in semantica/split/ner_extractors.py, then pass your method name during initialization.

Does chunk_overlap respect entity boundaries?

The overlap mechanism operates on finalized chunks, not during the initial boundary-aware construction. Overlap regions may contain partial entities from adjacent chunks. For strict entity isolation across overlaps, post-process chunks to deduplicate or mask cross-boundary entities.

How does this compare to semantic chunking without NER?

Standard semantic chunkers split at sentence or paragraph boundaries without entity knowledge. This risks cutting multi-word entities (organizations, persons, locations) across chunks, degrading GraphRAG relationship extraction. EntityAwareChunker eliminates this failure mode with minimal computational overhead.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}