How TextSplitter Performs Entity-Aware Chunking for GraphRAG in Semantica

Semantica’s TextSplitter uses named-entity recognition to map entity boundaries and prevents cuts across detected entities, ensuring each chunk contains complete person, location, and organization names that align with knowledge-graph nodes for GraphRAG.

The TextSplitter class in the semantica-agi/semantica repository serves as the unified entry point for document chunking, offering specialized strategies for knowledge-graph applications. When configured for entity-aware chunking, the splitter analyzes text using NERExtractor to preserve semantic units, producing chunks that maintain entity integrity critical for GraphRAG pipelines.

The Entity-Aware Chunking Pipeline

Named-Entity Recognition with NERExtractor

According to semantica/split/methods.py, the process begins by instantiating a NERExtractor with the selected ner_method parameter—either "ml" for traditional machine learning models or "llm" for language-model extraction. The extractor’s extract(text) method returns entity objects containing start_char and end_char positions that precisely locate each mention within the source document (lines 78-87).

Building the Entity-Boundary Map

The splitter aggregates these offsets into a set named entity_boundaries, inserting both the start and end character positions of every detected entity (lines 84-88). This lookup structure enables O(1) checks to determine whether any given text slice intersects with an entity mention.

Boundary-Respecting Split Logic

The core splitting algorithm follows a sentence-aware, boundary-respecting approach:

1. Sentence Tokenization: The document is first segmented using _split_sentences_regex to obtain natural linguistic units.
2. Boundary Checking: For each sentence, the algorithm invokes has_entity_boundary to verify if the sentence overlaps any entry in the entity_boundaries set.
3. Conditional Chunking: When accumulating sentences would exceed chunk_size and preserve_entities is True, the splitter prevents cuts across entity-containing sentences. Instead of splitting mid-entity, it carries the sentence over to the next chunk, ensuring complete entities remain within single chunks (lines 95-115).

This mechanism guarantees that person names, locations, and organizations remain undivided, allowing downstream GraphRAG systems to link entities across chunks without encountering fragmented references.

Fallback and Error Handling

If the optional semantic-extract package is unavailable, the implementation gracefully degrades to recursive character splitting (lines 72-76). This fallback ensures pipeline continuity even when NER dependencies are missing, though without entity-aware guarantees.

Implementation Code Examples

Basic entity-aware chunking using the ML NER method:

from semantica.split import TextSplitter

text = """
Apple announced the new iPhone 15 in California. Tim Cook said the device
features an improved camera and a faster chip.
"""

splitter = TextSplitter(
    method="entity_aware",
    ner_method="ml",          # Built-in ML NER

    chunk_size=200,
    preserve_entities=True,
)

chunks = splitter.split(text)
for i, c in enumerate(chunks, 1):
    print(f"Chunk {i}: {c.text!r}")

Advanced configuration with LLM-based entity extraction and batch processing:

splitter = TextSplitter(
    method="entity_aware",
    ner_method="llm",
    ner_kwargs={"model": "gpt-4o-mini"},
    chunk_size=500,
)

chunks = splitter.split_batch([doc1, doc2, doc3])

Key Source Files

• semantica/split/splitter.py: Public facade (TextSplitter) that forwards calls to the selected method implementation.
• semantica/split/methods.py: Contains split_entity_aware and the boundary-respecting logic (lines 52-115).
• semantica/split/kg_chunkers.py: Wrapper classes (EntityAwareChunker, RelationAwareChunker) providing KG-specific APIs.
• tests/split/test_chunkers.py: Unit tests verifying entity-aware behavior and GraphRAG integration.

Summary

• Entity Preservation: TextSplitter uses NERExtractor to map entity positions and stores them in an entity_boundaries set for fast lookup.
• Boundary-Respecting Splits: The algorithm prevents cuts across entity-containing sentences when preserve_entities=True, maintaining semantic integrity.
• GraphRAG Alignment: Whole entities within chunks directly correspond to knowledge-graph nodes, enabling accurate cross-document entity linking.
• Graceful Degradation: Falls back to recursive splitting if the semantic-extract dependency is missing.

Frequently Asked Questions

What NER methods does TextSplitter support for entity-aware chunking?

TextSplitter supports two primary ner_method values: "ml" for traditional machine learning NER models and "llm" for language-model-based extraction. When using "llm", you can pass additional configuration via ner_kwargs (e.g., {"model": "gpt-4o-mini"}) to specify the underlying model.

How does entity-aware chunking improve GraphRAG performance?

By preventing chunks from splitting entities like person names or organizations, entity-aware chunking ensures that vector embeddings represent complete semantic units. This alignment allows the knowledge graph to maintain consistent node identities across chunks, improving retrieval accuracy when linking entities in GraphRAG pipelines.

What happens if an entity is longer than the specified chunk_size?

If a single entity exceeds the chunk_size limit, the boundary-respecting logic cannot include the entity in a single chunk without violating size constraints. In such cases, the splitter prioritizes the chunk_size limit and may split the entity, though this scenario typically indicates that the chunk_size parameter should be increased for the specific document type.

Where is the core entity-aware splitting logic implemented?

The primary implementation resides in semantica/split/methods.py within the split_entity_aware function (lines 52-115). This module handles NER extraction, boundary map construction, and the sentence-accrual loop that respects entity positions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}