# How Entity Extraction in RAGAnything Builds the Multimodal Knowledge Graph

> Discover how RAGAnything's entity extraction builds a multimodal knowledge graph by structuring text, images, tables, and equations for enhanced RAG.

- Repository: [✨Data Intelligence Lab@HKU✨/RAG-Anything](https://github.com/HKUDS/RAG-Anything)
- Tags: internals
- Published: 2026-04-22

---

**Entity extraction in RAGAnything transforms raw text, images, tables, and equations into structured graph nodes and edges, then wires them together with "belongs_to" relationships to create a unified multimodal knowledge graph for retrieval-augmented generation.**

RAGAnything, developed by HKUDS, extends LightRAG's graph-based architecture to handle multimodal content. At the core of this system lies **entity extraction**—the process that identifies concepts from any modality and embeds them into a queryable knowledge structure. This article examines how RAGAnything's entity extraction pipeline operates, from chunk creation through graph persistence, using the actual source code implementation.

---

## From Raw Content to LightRAG Chunks

Before entity extraction can occur, RAGAnything converts each piece of multimodal content into a **LightRAG chunk**—a self-contained unit of data that can be processed independently.

In [`raganything/processor.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/processor.py), the system handles this through methods like `_process_multimodal_content_*`. Each chunk receives:

- A unique ID generated via `compute_mdhash_id` from [`raganything/utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/utils.py)
- Content text (e.g., image captions, table descriptions, or equation representations)
- Metadata including token count, chunk order, and source file path

These chunks are stored in LightRAG's `text_chunks` vector database (VDB), making them available for both retrieval and downstream entity extraction.

---

## Entity Extraction: The Core Pipeline

The heart of RAGAnything's knowledge graph construction occurs in `_batch_extract_entities_lightrag_style_type_aware`, located at **lines 1240-1268** of [`raganything/processor.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/processor.py).

### How the Extraction Method Works

This method receives a batch of chunks and orchestrates entity and relationship extraction through LightRAG's `extract_entities` function:

```python

# Simplified conceptual flow from raganything/processor.py

async def _batch_extract_entities_lightrag_style_type_aware(self, chunks):
    # chunks: dict of {chunk_id: chunk_data}

    
    # Delegate to LightRAG's extraction logic

    extraction_results = await self.lightrag.extract_entities(
        chunks, 
        # ... additional parameters for type-aware processing

    )
    
    # Returns list of (nodes, edges) tuples

    return extraction_results

```

The `extract_entities` function operates by:

1. **Running the LLM** on each chunk's content with a specialized prompt that asks for entity and relationship identification
2. **Parsing the LLM response** to extract structured information
3. **Returning nodes** (discovered entities with their types and descriptions) and **edges** (relationships between entities found within the same chunk)

---

## Wiring Entities to Their Parent Modalities with "belongs_to" Edges

Raw entity extraction produces isolated nodes and edges. RAGAnything's critical innovation is **linking every extracted entity back to its source modality** through a `"belongs_to"` relationship.

### The Belongs-To Relationship Logic

This occurs in `_batch_add_belongs_to_relations_type_aware` at **lines 1269-1310** of [`raganything/processor.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/processor.py):

```python

# Conceptual flow from the implementation

async def _batch_add_belongs_to_relations_type_aware(
    self, 
    chunk_results, 
    multimodal_data_list
):
    enhanced_results = []
    
    for (nodes, edges), modal_data in zip(chunk_results, multimodal_data_list):
        # Create "belongs_to" edges linking each entity to parent modality

        new_edges = []
        for node in nodes:
            belongs_to_edge = {
                "src_id": node["entity_name"],
                "tgt_id": modal_data["modal_entity_id"],  # e.g., "Figure 1 (image)"

                "edge_type": "belongs_to",
                "description": f"Entity belongs to {modal_data['modal_type']}"
            }
            new_edges.append(belongs_to_edge)
        
        enhanced_results.append((nodes, edges + new_edges))
    
    return enhanced_results

```

This same logic appears at the lower level in [`modalprocessors.py`](https://github.com/HKUDS/RAG-Anything/blob/main/modalprocessors.py) **lines 751-795**, where `_process_chunk_for_extraction` handles single-chunk processing with identical belongs-to wiring.

### Why "belongs_to" Matters

The `"belongs_to"` edge creates a **containment hierarchy** in the knowledge graph:

- **Entity nodes** represent fine-grained concepts ("solar system", " Jupiter", "chemical formula")
- **Modal entity nodes** represent the containing documents, images, tables, or equations
- **"belongs_to" edges** explicitly link concepts to their sources

This structure enables precise provenance tracking and modality-aware retrieval—queries can filter by source type or traverse from specific figures to their contained concepts.

---

## Graph Merging and Global Knowledge Assembly

After entity extraction and belongs-to wiring, RAGAnything merges all results into LightRAG's global knowledge graph through `merge_nodes_and_edges`.

### The Merging Process

```python

# Conceptual flow from the implementation

async def finalize_graph_update(self, all_chunk_results):
    all_nodes = []
    all_edges = []
    
    for nodes, edges in all_chunk_results:
        all_nodes.extend(nodes)
        all_edges.extend(edges)
    
    # Merge into global graph and vector stores

    await self.lightrag.merge_nodes_and_edges(all_nodes, all_edges)

```

This operation:

1. **Deduplicates entities** across chunks using semantic similarity and entity name matching
2. **Consolidates relationships** between identical entity pairs
3. **Updates vector stores** (`entities_vdb`, `relationships_vdb`) with new embeddings
4. **Maintains the graph structure** in a queryable format

### The Resulting Multimodal Knowledge Graph

The final graph contains:

| Component | Description | Example |
|-----------|-------------|---------|
| **Entity nodes** | Concepts from any modality | "photosynthesis", "bar chart", "CO₂" |
| **Modal entity nodes** | Container objects | "Figure 1 (image)", "Table 3 (table)", "Equation 2 (equation)" |
| **"belongs_to" edges** | Containment relationships | "photosynthesis" → "Figure 1 (image)" |
| **Semantic edges** | LLM-inferred relationships | "photosynthesis" → "related_to" → "chlorophyll" |

---

## Persistence and Queryability

The completed multimodal knowledge graph persists across LightRAG's storage layers:

- **`text_chunks`** — Original chunk content for retrieval
- **`entities_vdb`** — Vector embeddings of all entity nodes
- **`relationships_vdb`** — Vector embeddings of relationship descriptions
- **Graph store** — Structured node and edge data for traversal queries

This persistence enables cross-modal reasoning: a query about "solar system diagrams" can retrieve relevant images through their "belongs_to" edges, then traverse to connected entities like "Jupiter" or "asteroid belt" regardless of original modality.

---

## Summary

Entity extraction in RAGAnything serves as the critical bridge between unstructured multimodal content and structured knowledge representation:

- **Chunking** converts diverse content into processable units stored in `text_chunks`
- **Batch extraction** via `_batch_extract_entities_lightrag_style_type_aware` discovers entities and relationships using LLM inference
- **Belongs-to wiring** through `_batch_add_belongs_to_relations_type_aware` links every entity to its parent modality, creating provenance trails
- **Graph merging** consolidates all results into LightRAG's unified knowledge structure
- **Persistence** in vector and graph stores enables sophisticated cross-modal retrieval

The result is a **multimodal knowledge graph** where concepts from text, images, tables, and equations coexist as interconnected entities—enabling retrieval-augmented generation that reasons across traditional modality boundaries.

---

## Frequently Asked Questions

### What makes RAGAnything's entity extraction "multimodal"?

RAGAnything treats images, tables, equations, and text as equal citizens in the knowledge graph. Each modality becomes a **modal entity** node, and entity extraction runs on descriptive representations of each (e.g., image captions, table summaries). The `"belongs_to"` edges then link extracted concepts back to their visual or structural sources, enabling queries that understand which figure contains which concept.

### How does the "belongs_to" relationship improve retrieval accuracy?

Without `"belongs_to"`, extracted entities float disconnected from their origins. The `"belongs_to"` edge creates **provenance chains**: when retrieving "Jupiter," the system knows it came from "Figure 1 (image)" in "solar_system.pdf." This enables **source-grounded answers** (citing specific figures) and **modality filtering** (restricting searches to tables or images only).

### Can I extract entities from a single image without processing an entire document?

Yes. The lower-level API in [`modalprocessors.py`](https://github.com/HKUDS/RAG-Anything/blob/main/modalprocessors.py) supports single-chunk processing. Use `_process_chunk_for_extraction` (lines 751-795) directly, or construct a LightRAG-style chunk manually and pass it to `Processor._batch_extract_entities_lightrag_style_type_aware`. This is useful for streaming scenarios or when integrating with external image captioning pipelines.

### What LLM prompts does RAGAnything use for entity extraction?

RAGAnything delegates to LightRAG's `extract_entities`, which constructs prompts requesting: (1) **entity names** with types and descriptions, (2) **relationships** between entities found in the same chunk, with source entity, target entity, relation type, and description. The prompts are modality-agnostic—image captions, table summaries, and raw text all feed into the same extraction logic, with modality context preserved through the `"belongs_to"` wiring step.