# How LightRAG Performs Knowledge Graph Extraction and Entity Relationship Building

> Discover how LightRAG constructs knowledge graphs. Learn about its three-stage pipeline for entity extraction, relationship building, and persistent storage.

- Repository: [✨Data Intelligence Lab@HKU✨/LightRAG](https://github.com/HKUDS/LightRAG)
- Tags: how-to-guide
- Published: 2026-03-23

---

**LightRAG constructs knowledge graphs through a three-stage pipeline that chunks documents, uses cached LLM calls to extract structured entities and relationships via custom delimiters, and atomically persists results to graph and vector storage.**

LightRAG is an open-source retrieval-augmented generation framework developed by HKUDS that transforms unstructured text into queryable knowledge graphs. The system implements a deterministic extraction pipeline in Python that bridges large language models with persistent graph storage backends like Neo4j, OpenSearch, and NetworkX. Understanding how LightRAG performs knowledge graph extraction and entity relationship building reveals the architectural decisions that enable high-throughput, concurrent graph construction.

## The Three-Stage Knowledge Graph Extraction Pipeline

The extraction process in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py) and [`lightrag/lightrag.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/lightrag.py) follows a tightly-coupled workflow that converts raw text into persistent graph structures through chunking, parsing, and atomic storage operations.

### Stage 1: Chunking and LLM Prompting

Documents enter the pipeline through `chunking_by_token_size` in [`utils.py`](https://github.com/HKUDS/LightRAG/blob/main/utils.py), which splits text into token-limited segments. The `extract_entities` function (lines 13-28 in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py)) prepares specialized prompts using templates from [`prompt.py`](https://github.com/HKUDS/LightRAG/blob/main/prompt.py):

- **System prompt**: Defines the extraction format and entity/relationship schema
- **User prompt**: Contains the chunked text content

Each chunk is processed through `use_llm_func_with_cache` to avoid redundant LLM calls, storing responses with `cache_type="extract"` keyed by `chunk_id`. This caching layer ensures idempotent processing during re-indexing operations.

### Stage 2: Parsing Delimited LLM Output

Raw LLM responses follow a strict delimiter protocol using `<|#|>` as the tuple separator and `<|COMPLETE|>` as the completion marker. The `_process_extraction_result` function (lines 279-340 in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py)) orchestrates parsing:

1. **Splitting**: Separates the response string on custom delimiters
2. **Entity extraction**: `_handle_single_entity_extraction` extracts `entity_name`, `entity_type`, and `description`, applying normalization and sanitization via `sanitize_and_normalize_extracted_text`
3. **Relationship extraction**: `_handle_single_relationship_extraction` extracts `src_id`, `tgt_id`, `description`, `keywords`, and optional weight values

Malformed records are discarded during parsing, while valid entries are converted into Python dictionaries ready for storage.

### Stage 3: Graph Construction and Atomic Persistence

The orchestrator `_process_extract_entities` in [`lightrag/lightrag.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/lightrag.py) (lines 91-128) routes parsed data to storage:

- **Graph storage**: Calls `upsert_node` and `upsert_edge` on `BaseGraphStorage` instances
- **Vector storage**: Generates IDs using `compute_mdhash_id` with `ent-` and `rel-` prefixes, then upserts embedding-ready documents to entity and relationship vector databases
- **Chunk tracking**: Optional KV stores (`entity_chunks_storage`, `relation_chunks_storage`) maintain provenance links between graph elements and source text chunks

Updates are flushed atomically via `_persist_graph_updates` in [`lightrag/utils_graph.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/utils_graph.py) (lines 23-64), which executes `index_done_callback` across all storage objects in parallel to ensure consistency.

## Multi-Pass Extraction and Gleaning Logic

When `entity_extract_max_gleaning` exceeds zero, LightRAG initiates continuation prompts to extract additional entities missed in the first pass. The gleaning mechanism (lines 2950-2990 in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py)) sends a "continue extraction" prompt to the LLM, parses the supplementary results, and merges them with the initial extraction by keeping the longer description for duplicate entities or relationships. This iterative approach improves recall on dense documents without exponential token cost increases.

## Practical Implementation Examples

### Basic End-to-End Document Indexing

```python
from lightrag import LightRAG
from pathlib import Path

# Initialize the engine with configured storage backends

rag = LightRAG()

# Ingest documents - automatically handles chunking and extraction

doc_path = Path("data/research_papers.txt")
await rag.insert_documents([doc_path])

# Query extracted entities

entity_data = await rag.get_entity("Machine Learning")
print(entity_data["description"])

```

### Manual Single-Chunk Extraction

```python
from lightrag.operate import extract_entities
from lightrag.utils import Tokenizer

# Simulate chunked input

chunks = {
    "chunk-001": {
        "content": "Neural networks are computational models inspired by biological brains.",
        "tokens": 12,
        "full_doc_id": "doc-abc",
        "chunk_order_index": 0,
    }
}

global_config = {
    "llm_model_func": openai_chat_completion,
    "tokenizer": Tokenizer(),
    "entity_extract_max_gleaning": 1,
    "max_extract_input_tokens": 8192,
    "addon_params": {"language": "en"},
}

# Returns list of dicts with maybe_nodes and maybe_edges

results = await extract_entities(chunks, global_config)
print(results[0]["maybe_nodes"])

```

### Direct Graph Manipulation

```python
from lightrag.utils_graph import acreate_entity, acreate_relation
from lightrag.base import get_graph_storage, get_vector_storage

# Initialize storage adapters

graph = await get_graph_storage("neo4j")
ent_vdb = await get_vector_storage("faiss")
rel_vdb = await get_vector_storage("faiss")

# Create entity node

await acreate_entity(
    chunk_entity_relation_graph=graph,
    entities_vdb=ent_vdb,
    relationships_vdb=rel_vdb,
    entity_name="Transformer Architecture",
    entity_data={"description": "Attention-based neural architecture", "entity_type": "model"},
)

# Create relationship edge

await acreate_relation(
    chunk_entity_relation_graph=graph,
    entities_vdb=ent_vdb,
    relationships_vdb=rel_vdb,
    source_entity="Transformer Architecture",
    target_entity="Attention Mechanism",
    relation_data={"description": "Utilizes", "keywords": "attention,self-attention"},
)

```

## Summary

- **Three-stage pipeline**: LightRAG extracts knowledge graphs through chunking, LLM-based extraction with custom delimiters (`<|#|>` and `<|COMPLETE|>`), and atomic persistence via [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py) and [`lightrag/utils_graph.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/utils_graph.py).
- **Robust parsing**: The `_handle_single_entity_extraction` and `_handle_single_relationship_extraction` functions sanitize and validate LLM outputs before storage, discarding malformed records.
- **Caching architecture**: `use_llm_func_with_cache` prevents duplicate LLM calls during reprocessing, keyed by chunk identifiers and extraction type.
- **Gleaning support**: Multi-pass extraction with `entity_extract_max_gleaning` improves recall by continuing extraction until no new entities are found, merging results intelligently.
- **Atomic updates**: `_persist_graph_updates` ensures graph storage, vector databases, and chunk tracking KV stores commit simultaneously via parallel callbacks to prevent partial writes.

## Frequently Asked Questions

### What delimiter format does LightRAG expect for entity extraction?

LightRAG uses two specific delimiters in the LLM prompt output: `<|#|>` separates fields within entity and relationship tuples, while `<|COMPLETE|>` marks the end of extraction. These delimiters are defined in [`lightrag/constants.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/constants.py) and parsed by `_process_extraction_result` in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py). The system includes helper functions to clean corrupted delimiters before parsing, ensuring robust extraction even with imperfect LLM outputs.

### How does LightRAG handle concurrent graph updates?

The framework implements atomic persistence through `_persist_graph_updates` in [`lightrag/utils_graph.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/utils_graph.py), which executes `index_done_callback` across all storage backends (graph, vector, and KV stores) in parallel. This design prevents partial writes during high-concurrency indexing operations, ensuring that entity nodes, relationship edges, and their corresponding embeddings commit together or roll back together, maintaining graph consistency across Neo4j, OpenSearch, or other supported backends.

### What is "gleaning" in LightRAG's extraction process?

Gleaning refers to LightRAG's multi-pass extraction strategy where, after the initial LLM call, the system sends a continuation prompt asking the model to extract any additional entities or relationships it may have missed. When `entity_extract_max_gleaning` is configured greater than zero, the pipeline parses these supplementary results and merges them with the initial extraction in [`lightrag/operate.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/operate.py), keeping the longer description when duplicates are detected. This improves extraction completeness on information-dense documents without requiring excessive token consumption.

### Which storage backends support LightRAG's knowledge graph storage?

LightRAG abstracts graph storage through `BaseGraphStorage` and currently supports Neo4j, OpenSearch, and NetworkX backends for the graph structure. For vector storage of entity and relationship embeddings, the system supports Qdrant, Milvus, Faiss, and other vector databases. The storage adapters are selected during `LightRAG` initialization and are invoked through `upsert_node` and `upsert_edge` methods defined in [`lightrag/lightrag.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/lightrag.py), allowing flexible deployment across local or distributed infrastructure.