How to Configure chunk_token_size and chunk_overlap_token_size in LightRAG

LightRAG exposes chunk_token_size (default 1200) and chunk_overlap_token_size (default 100) through environment variables (CHUNK_SIZE, CHUNK_OVERLAP_SIZE), CLI flags (--chunk-size, --chunk-overlap-size), and Python constructor arguments to control text granularity and cross-chunk context preservation.

LightRAG splits input documents into token-sized chunks before embedding generation and knowledge graph insertion. The chunk_token_size parameter defines the maximum tokens per segment, while chunk_overlap_token_size specifies how many tokens repeat across adjacent chunks to preserve semantic continuity. Proper configuration of these parameters lets you optimize retrieval accuracy, manage embedding API costs, and respect your vector store's token limits.

Understanding the Text Splitting Parameters

Two dataclass fields in lightrag/lightrag.py govern document segmentation:

• chunk_token_size: The hard limit on tokens per chunk. When a document exceeds this value, LightRAG divides it into consecutive segments. The default is 1200 tokens, read from the CHUNK_SIZE environment variable if not provided explicitly.
• chunk_overlap_token_size: The number of tokens shared between consecutive chunks. This overlap ensures that phrases crossing a chunk boundary appear in both chunks, improving recall during retrieval. The default is 100 tokens, read from CHUNK_OVERLAP_SIZE.

The built-in splitter chunking_by_token_size (referenced via LightRAG.chunking_func receives these values as max_tokens and overlap, applying them after tokenization.

Configuration Methods

LightRAG supports three configuration layers, allowing you to set defaults globally or override them per instance.

Environment Variables

Set process-wide defaults that apply to all LightRAG instances:

export CHUNK_SIZE=800
export CHUNK_OVERLAP_SIZE=200
python -m lightrag.server --working-dir ./rag

These variables act as fallbacks when you do not provide explicit Python arguments or CLI flags.

Python API Constructor

Override defaults for a specific instance by passing arguments directly to the LightRAG class:

from lightrag import LightRAG

rag = LightRAG(
    working_dir="my_rag",
    llm_model_func=my_llm,
    embedding_func=my_embed,
    chunk_token_size=512,
    chunk_overlap_token_size=256
)
await rag.initialize_storages()

Constructor arguments take precedence over environment variables.

Command Line Interface

When running the LightRAG server, the parser in lightrag/api/config.py exposes dedicated flags:

python -m lightrag.server \
  --working-dir ./rag \
  --chunk-size 600 \
  --chunk-overlap-size 150 \
  --llm-binding openai \
  --embedding-binding openai

The configuration parser maps --chunk-size to args.chunk_size and --chunk-overlap-size to args.chunk_overlap_size, injecting these values into the LightRAG dataclass during startup.

Implementation and Runtime Behavior

Tokenization and Fallbacks

The default chunking_by_token_size function tokenizes content using tiktoken (or your configured tokenizer), then slices the token array into chunks of max_tokens length, stepping forward by max_tokens - overlap tokens each iteration. If the tokenizer fails to load, the function falls back to character-based splitting while still respecting the overlap size constraint.

Dynamic Query-Time Adjustment

During retrieval operations, LightRAG calculates an available_chunk_tokens budget based on the query context and LLM token limits. As implemented in lightrag/operate.py (lines 4053-4068), this dynamic value passes as chunk_token_limit to the chunking routine, potentially reducing the effective chunk size below your configured chunk_token_size to prevent context window overflow.

Custom Chunking Functions

For domain-specific splitting (e.g., by paragraphs or semantic boundaries), provide a callable matching the expected signature:

def semantic_chunker(tokenizer, content, split_by_char, split_by_char_only,
                     overlap, max_tokens):
    """Preserve paragraph boundaries where possible."""
    paragraphs = content.split("\n\n")
    chunks = []
    current_tokens = []
    
    for para in paragraphs:
        para_tokens = tokenizer.encode(para)
        if len(current_tokens) + len(para_tokens) > max_tokens and current_tokens:
            # Flush current buffer

            chunks.append(tokenizer.decode(current_tokens))
            # Carry over overlap

            overlap_start = max(0, len(current_tokens) - overlap)
            current_tokens = current_tokens[overlap_start:] + para_tokens
        else:
            current_tokens.extend(para_tokens)
    
    if current_tokens:
        chunks.append(tokenizer.decode(current_tokens))
    
    return [{"content": c, "tokens": len(tokenizer.encode(c)), 
             "chunk_order_index": i} for i, c in enumerate(chunks)]

rag = LightRAG(
    working_dir="rag",
    chunk_token_size=1024,
    chunk_overlap_token_size=64,
    chunking_func=semantic_chunker
)

Your custom function receives max_tokens and overlap as configured, allowing you to implement alternative logic while respecting the user-specified constraints.

Summary

• Parameter defaults: chunk_token_size defaults to 1200 (env: CHUNK_SIZE); chunk_overlap_token_size defaults to 100 (env: CHUNK_OVERLAP_SIZE).
• Configuration hierarchy: Constructor arguments override CLI flags, which override environment variables.
• Source files: Definitions reside in lightrag/lightrag.py, CLI parsing in lightrag/api/config.py, and dynamic limits in lightrag/operate.py.
• Chunking function: Default chunking_by_token_size handles tokenization; fallback to character-based splitting occurs if tiktoken is unavailable.
• Query adaptation: Runtime token budgeting may temporarily reduce chunk size during retrieval to fit within LLM context windows.

Frequently Asked Questions

What happens if chunk_token_size exceeds my embedding model's limit?

LightRAG passes the full chunk to your configured embedding_func. If the chunk exceeds the model's maximum input tokens, the embedding API will likely truncate the text or return an error. Keep chunk_token_size within your specific model's limit (commonly 512, 1024, or 2048 tokens).

Does increasing chunk_overlap_token_size consume more storage?

Yes. Overlap causes redundant token storage across adjacent chunks, increasing the total number of embeddings generated per document. While this improves retrieval recall by preserving boundary context, it increases vector database storage costs and insertion time.

Can I modify chunk sizes for documents already in storage?

These parameters only affect new ingestion operations. Existing chunks remain at their original sizes. To apply new settings, clear your working directory or storage backend and re-process your documents.

How does LightRAG handle documents smaller than chunk_token_size?

Documents containing fewer tokens than chunk_token_size are stored as single chunks. The chunk_overlap_token_size parameter only applies when content requires splitting into multiple segments.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}