How to Troubleshoot LightRAG Entity Extraction Issues When the Knowledge Graph Is Not Building

When LightRAG fails to populate the knowledge graph, the issue typically stems from the token-count guard blocking extraction gleaning, malformed LLM output that breaks the tuple parser, or unconfigured storage objects that skip persistence calls.

LightRAG is a modular retrieval-augmented generation framework that constructs a knowledge graph by extracting entities and relationships from text chunks using LLM calls. When the graph remains empty after processing documents, the failure usually occurs silently during the extract_entities workflow in lightrag/operate.py. This guide examines the exact source locations—from line 2813 where the chunk loop starts to the _persist_graph_updates utility—providing specific diagnostics to identify and resolve extraction failures.

How LightRAG Extracts Entities to Build the Knowledge Graph

The extraction process follows a strict pipeline defined in lightrag/operate.py. Understanding these steps is essential for locating where the process silently drops data.

• Chunk iteration (line 2813): The extract_entities function receives a dictionary of TextChunkSchema objects and iterates over each chunk.
• Initial LLM extraction: For each chunk, it calls use_llm_func_with_cache with the entity_extraction_user_prompt, then parses the result via _process_extraction_result.
• Optional gleaning loop (lines 2816-2845): If entity_extract_max_gleaning is greater than 0 and the token count is under max_extract_input_tokens, a second LLM request is issued using entity_continue_extraction_user_prompt. The token guard at lines 2830-2835 prevents gleaning when input exceeds the limit.
• Result merging (lines 2850-2875): Gleaned entities and relations replace originals only if they contain longer descriptions.
• Persistence: Finally, _persist_graph_updates in lightrag/utils_graph.py writes vectors, graph edges, and chunk-tracking data to storage.

If any step returns empty or malformed data without raising an exception, the downstream graph will appear empty.

Common Reasons the Knowledge Graph Stays Empty

Several specific misconfigurations cause the extraction pipeline to return empty results. Check these symptoms against your logs and configuration.

Symptom: No LLM calls after the first request

• Cause: entity_extract_max_gleaning is set to 0 in your global configuration.
• Fix: Verify global_config["entity_extract_max_gleaning"] > 0 to enable multi-pass extraction.

Symptom: "Gleaning stopped" warning appears

• Cause: The token-count guard at lines 2833-2837 in operate.py halts processing because token_count > max_extract_input_tokens.
• Fix: Increase max_extract_input_tokens or reduce chunk size to stay within limits.

Symptom: LLM returns malformed tuples

• Cause: The extraction output does not match the expected delimiters. The parser expects tuples formatted as (entity<|#|>NAME<|#|>TYPE<|#|>DESC)<|COMPLETE|> using PROMPTS["DEFAULT_TUPLE_DELIMITER"] (<|#|>) and DEFAULT_COMPLETION_DELIMITER (<|COMPLETE|>).
• Fix: Inspect raw LLM output and verify delimiter alignment.

Symptom: _process_extraction_result raises exceptions

• Cause: Invalid JSON or missing fields in the LLM output cause the parser to fail.
• Fix: Wrap extract_entities in try/except blocks and run unit tests in tests/test_extract_entities.py to isolate malformed responses.

Symptom: Graph storage not updated

• Cause: chunk_entity_relation_graph, entities_vdb, or relationships_vdb are None or do not implement BaseKGStorage / BaseVectorStorage from lightrag/base.py.
• Fix: Confirm storage objects are instantiated before calling extraction.

Symptom: Deadlock during processing

• Cause: The get_storage_keyed_lock function receives wrong namespace or duplicate keys.
• Fix: Enable DEBUG logging for lightrag.utils and verify "acquired lock" messages appear consistently.

Step-by-Step Troubleshooting Workflow

Run this checklist in a Python REPL or test script to isolate the failure point. Each step references specific source lines for verification.

Inspect Global Configuration

Start by verifying your token limits and gleaning settings.

from lightrag.utils import Tokenizer

class DummyTokenizer:
    def encode(self, txt): return [ord(c) for c in txt]
    def decode(self, toks): return "".join(chr(t) for t in toks)

cfg = {
    "llm_model_func": AsyncMock(return_value="…"),
    "entity_extract_max_gleaning": 1,
    "max_extract_input_tokens": 10,   # tiny to trigger guard

    "tokenizer": Tokenizer("dummy", DummyTokenizer()),
    "addon_params": {"language": "en"},
}
print(cfg["entity_extract_max_gleaning"], cfg["max_extract_input_tokens"])

Reference: See tests/test_extract_entities.py in the _make_global_config helper (lines 20-33).

Run Extraction with Limited Tokens

Force the token guard to fire and observe the warning.

chunks = {
    "chunk-001": {
        "content": "Some long text that certainly pushes the token count beyond ten tokens…",
        "tokens": 200,
        "full_doc_id": "doc-001",
        "chunk_order_index": 0,
    }
}

await extract_entities(
    chunks=chunks,
    global_config=cfg,
)

Expect a single LLM call and a warning logged at operate.py lines 2833-2836.

Check for the Warning Message

The warning contains the phrase "Gleaning stopped … exceeded limit". If this message is absent, the guard is either bypassed due to configuration errors or your logger level is muted.

Increase Token Limits and Verify Gleaning

Raise the token budget to allow the second LLM call.

cfg["max_extract_input_tokens"] = 999_999
await extract_entities(chunks=chunks, global_config=cfg)

The test test_gleaning_proceeds_when_tokens_within_limit expects await_count == 2 on the LLM mock.

Validate LLM Output Format

Manually test the parser with correctly formatted tuples.

result = "(entity<|#|>TEST<|#|>CONCEPT<|#|>A test entity)<|COMPLETE|>"
nodes, edges = await _process_extraction_result(
    result, "chunk-001", time.time(), "demo.py",
    tuple_delimiter="<|#|>", completion_delimiter="<|COMPLETE|>"
)
print(nodes)   # should contain a dict entry for "TEST"

The parser logic resides in lightrag/operate.py around line 2910.

Confirm Persistence

Verify that storage callbacks are invoked after extraction.

class DummyStorage:
    async def index_done_callback(self): 
        print("persisted")

await extract_entities(
    chunks=chunks,
    global_config=cfg,
    chunk_entity_relation_graph=DummyStorage(),
    entities_vdb=DummyStorage(),
    relationships_vdb=DummyStorage(),
)

Check that index_done_callback fires for each storage instance (see utils_graph.py lines 23-64).

Run Unit Tests

Execute the built-in test suite to verify core logic.

python -m pytest tests/test_extract_entities.py

All three tests must pass; failures indicate regressions in the token-guard logic.

Practical Code Examples for Debugging

These scripts demonstrate how to reproduce specific failure modes and verify fixes.

Reproducing the Token Guard Warning

This minimal script triggers the "Gleaning stopped" warning by setting a restrictive token limit.

import asyncio
from unittest.mock import AsyncMock
from lightrag.utils import Tokenizer
from lightrag.operate import extract_entities

class DummyTokenizer:
    def encode(self, txt): return [ord(c) for c in txt]
    def decode(self, toks): return "".join(chr(t) for t in toks)

global_cfg = {
    "llm_model_func": AsyncMock(
        return_value="(entity<|#|>E1<|#|>TYPE<|#|>Desc)<|COMPLETE|>"
    ),
    "entity_extract_max_gleaning": 1,
    "max_extract_input_tokens": 10,  # Exceeded by chunk text

    "tokenizer": Tokenizer("dummy", DummyTokenizer()),
    "addon_params": {"language": "en"},
}

chunks = {
    "chunk-001": {
        "content": "Some long text that certainly pushes the token count beyond ten tokens…",
        "tokens": 200,
        "full_doc_id": "doc-001",
        "chunk_order_index": 0,
    }
}

async def main():
    await extract_entities(chunks=chunks, global_config=global_cfg)

asyncio.run(main())

Running this produces the log warning:

WARNING:lightrag.operate:Gleaning stopped for chunk chunk-001: Input tokens (123) exceeded limit (10).

Disabling the Token Guard

When you know your model can handle the context size, effectively disable the guard by setting a high limit.

global_cfg["max_extract_input_tokens"] = 1_000_000
global_cfg["entity_extract_max_gleaning"] = 1

This configuration triggers two LLM calls (initial extraction plus gleaning). Verify this behavior with:

assert global_cfg["llm_model_func"].await_count == 2

Verifying Graph Persistence

Ensure that extraction results actually reach your storage backend.

class DummyStorage:
    async def index_done_callback(self):
        print("Graph data persisted")

dummy_graph = DummyStorage()
dummy_entities = DummyStorage()
dummy_relations = DummyStorage()

await extract_entities(
    chunks=chunks,
    global_config=global_cfg,
    chunk_entity_relation_graph=dummy_graph,
    entities_vdb=dummy_entities,
    relationships_vdb=dummy_relations,
)

Expected output: three "Graph data persisted" messages, confirming that _persist_graph_updates invoked each storage's callback.

Key Source Files to Inspect

Understanding these files helps you trace data flow during debugging:

• lightrag/operate.py (line 2813): Contains extract_entities, the token-guard logic, and the merging algorithm.
• lightrag/utils_graph.py (line 23): Implements _persist_graph_updates, which handles writing to vector and graph stores.
• tests/test_extract_entities.py: Unit tests covering token-guard behavior and gleaning logic.
• lightrag/utils.py: Defines the Tokenizer class used for the token-count estimates that trigger the guard.
• lightrag/base.py: Abstract base classes BaseKGStorage and BaseVectorStorage that your storage objects must implement.

Summary

• The token-count guard (lines 2830-2835) silently skips gleaning when chunks exceed max_extract_input_tokens, leaving extractions incomplete.
• Malformed LLM output that deviates from the expected (entity<|#|>...)<|COMPLETE|> format causes _process_extraction_result to return empty results.
• Storage misconfiguration—passing None or non-compliant objects to extract_entities—prevents _persist_graph_updates from persisting data.
• Enabling entity_extract_max_gleaning and verifying token limits are the first steps to ensuring complete entity extraction.
• Always validate extraction by running pytest tests/test_extract_entities.py and inspecting logs for the "Gleaning stopped" warning.

Frequently Asked Questions

Why does LightRAG extract entities but fail to build the knowledge graph?

LightRAG may successfully call the LLM but fail to populate the graph if the token-count guard halts gleaning or if the output parser encounters malformed tuples. Additionally, if chunk_entity_relation_graph, entities_vdb, or relationships_vdb are not properly initialized as BaseKGStorage or BaseVectorStorage instances, the _persist_graph_updates function cannot save the results, leaving the graph empty despite successful extraction.

How do I fix the "Gleaning stopped" warning in LightRAG?

Increase the max_extract_input_tokens value in your global configuration to accommodate your chunk sizes, or reduce the chunk size to stay below the token limit. The warning at lines 2833-2837 in lightrag/operate.py triggers when token_count > max_extract_input_tokens, preventing the second-pass extraction that captures additional entities.

What format does LightRAG expect from the LLM for entity extraction?

LightRAG expects tuples delimited by the string defined in PROMPTS["DEFAULT_TUPLE_DELIMITER"] (default <|#|>) and terminated by DEFAULT_COMPLETION_DELIMITER (default <|COMPLETE|>). A valid response looks like (entity<|#|>EntityName<|#|>EntityType<|#|>Description)<|COMPLETE|>. If your LLM returns JSON or uses different separators, _process_extraction_result will fail to parse the entities.

How can I test if my LightRAG storage configuration is correct?

Create mock storage classes that implement index_done_callback and pass them to extract_entities. If the callback fires after extraction, your configuration is wired correctly. Also verify that your storage objects inherit from BaseKGStorage for graph data and BaseVectorStorage for vector data as defined in lightrag/base.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}