How to Insert Pre-Parsed Content Lists Directly into RAGAnything Without Document Parsing

Use the insert_content_list method from ProcessorMixin to bypass document parsing and inject pre-processed content lists—containing text, images, tables, and equations—directly into the RAG pipeline.

The HKUDS/RAG-Anything repository provides a multimodal RAG framework that normally relies on document parsers to extract content before insertion. When you need to insert pre-parsed content lists directly into RAGAnything—whether from external OCR tools, curated datasets, or custom extraction pipelines—you can skip the parser entirely and use the direct insertion API to build vector stores and knowledge graphs immediately.

Understanding the Direct Insertion Pipeline

The direct insertion API treats your pre-parsed data as a first-class document source, processing it through the same embedding, chunking, and graph construction stages as parsed files, but without the overhead of format detection or extraction.

The insert_content_list Method Signature

Located in raganything/processor.py (lines 68–100), the ProcessorMixin.insert_content_list method accepts a structured content list and orchestrates the full insertion flow:

async def insert_content_list(
    self,
    content_list: list[dict],
    file_path: str | None = None,
    doc_id: str | None = None,
    display_stats: bool | None = None,
    **kwargs
) -> bool

Key parameters:

• content_list – A list of dictionaries containing text blocks, images, tables, or equations with metadata such as page_idx and captions.
• file_path – Optional source identifier used for citation purposes only; no parsing occurs.
• doc_id – Optional unique identifier; if omitted, _generate_content_based_doc_id creates a deterministic hash from the content (line 176–179).
• display_stats – Overrides the global display_content_stats config flag to log processing statistics.

How It Differs from Standard Document Parsing

Standard ingestion relies on insert_file or insert_directory methods that invoke format-specific parsers (PDF, DOCX, etc.) to generate content lists. By calling insert_content_list directly, you assume responsibility for structuring the data, while RAGAnything handles vectorization, multimodal captioning, and knowledge graph construction. According to the source code in raganything/raganything.py, the system ensures LightRAG initialization via _ensure_lightrag_initialized() (lines 55–61) before any insertion occurs, guaranteeing storage backends are ready regardless of entry point.

Step-by-Step Implementation Guide

1. Initialize RAGAnything with LightRAG Backend

Configure your instance with the necessary model functions and enable multimodal processing toggles. The RAGAnythingConfig class centralizes pipeline settings including enable_image_processing, enable_table_processing, and display_content_stats:

from raganything import RAGAnything, RAGAnythingConfig
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc
import os

def llm_model_func(prompt, system_prompt=None, history_messages=None, **kw):
    return openai_complete_if_cache(
        "gpt-4o-mini",
        prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        api_key=os.getenv("OPENAI_API_KEY"),
        **kw,
    )

embedding_func = EmbeddingFunc(
    embedding_dim=3072,
    max_token_size=8192,
    func=lambda txts: openai_embed.func(
        txts,
        model="text-embedding-3-large",
        api_key=os.getenv("OPENAI_API_KEY"),
    ),
)

config = RAGAnythingConfig(
    working_dir="./rag_storage",
    enable_image_processing=True,
    enable_table_processing=True,
    enable_equation_processing=True,
)

rag = RAGAnything(
    config=config,
    llm_model_func=llm_model_func,
    embedding_func=embedding_func,
)

2. Prepare Your Content List Structure

Construct a list where each element is a dictionary with a type field. Valid types include text, image, table, and equation. Each block should include page_idx for context tracking:

content_list = [
    {"type": "text", "text": "RAGAnything supports direct content injection.", "page_idx": 0},
    {
        "type": "image",
        "img_path": "/path/to/diagram.png",
        "image_caption": ["Figure 1: Architecture"],
        "image_footnote": ["System overview"],
        "page_idx": 1
    },
    {
        "type": "table",
        "table_body": "| Metric | Value |\n|---|---|\n| Latency | 200ms |",
        "table_caption": ["Performance metrics"],
        "page_idx": 2
    },
    {
        "type": "equation",
        "latex": r"\mathrm{Score} = \sum_i w_i \cdot \mathrm{sim}(q_i, d_i)",
        "text": "Similarity scoring formula",
        "page_idx": 3
    }
]

3. Execute the Insertion with Context Awareness

Call insert_content_list asynchronously. The method handles text extraction, multimodal processing, and callback dispatching automatically:

async def insert_data():
    await rag.insert_content_list(
        content_list=content_list,
        file_path="manual_entry.json",
        doc_id="custom-doc-001",
        display_stats=True,
    )
    
    # Query immediately after insertion

    result = await rag.aquery(
        "Explain the similarity scoring formula",
        mode="hybrid"
    )
    print(result)

import asyncio
asyncio.run(insert_data())

Deep Dive: Internal Processing Flow

Understanding the internal mechanics helps optimize your content list structure and troubleshoot insertion issues.

Content Separation and Statistics

Upon invocation, insert_content_list first checks display_stats against the global configuration (lines 68–76). If enabled, it logs a breakdown of block types (text, image, table, equation, custom) to the console (lines 220–237).

The separate_content(content_list) function (line 238–239) bifurcates the list into:

• Plain text blocks – Concatenated for standard LightRAG insertion.
• Multimodal blocks – Images, tables, and equations requiring specialized processing.

Multimodal Context Extraction

Before processing multimodal items, the method registers the entire content list as a context source (lines 242–247). This enables the context extractor (raganything/enhanced_markdown.py) to retrieve surrounding text for caption generation. Each multimodal block dispatches to its respective processor:

• ImageModalProcessor for images
• TableModalProcessor for tables
• EquationModalProcessor for mathematical expressions

These processors utilize surrounding textual context to generate rich embeddings and captions, ensuring the knowledge graph maintains semantic relationships between modalities.

Text Chunking and Knowledge Graph Construction

The extracted plain text undergoes insert_text_content (lines 251–274), which:

1. Applies character-based splitting if split_by_character is configured.
2. Stores chunks in LightRAG's vector store.
3. Triggers on_text_insert_start and on_text_insert_complete callbacks if a CallbackManager is present.

After multimodal processing completes, the document status is marked complete and on_document_complete fires, signaling that the pre-parsed content is fully integrated into the searchable knowledge graph.

Complete Working Example

The repository includes a ready-to-run demonstration in examples/insert_content_list_example.py. Below is an adapted version showing vision model integration:

import asyncio
import os
from dotenv import load_dotenv
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.llm.openai import openai_complete_if_cache
from lightrag.utils import logger

load_dotenv()

def vision_model_func(prompt, system_prompt=None, history_messages=None, 
                     image_data=None, **kw):
    if image_data:
        return openai_complete_if_cache(
            "gpt-4o",
            "",
            system_prompt=system_prompt,
            messages=[
                {"role": "user", "content": [
                    {"type": "text", "text": prompt},
                    {"type": "image_url", 
                     "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}
                ]}
            ],
            api_key=os.getenv("OPENAI_API_KEY"),
            **kw,
        )
    return openai_complete_if_cache(
        "gpt-4o-mini", prompt, system_prompt=system_prompt, **kw
    )

# Initialize with multimodal support

rag = RAGAnything(
    config=RAGAnythingConfig(
        working_dir="./demo_storage",
        enable_image_processing=True,
        display_content_stats=True,
    ),
    llm_model_func=lambda **kw: openai_complete_if_cache(
        "gpt-4o-mini", api_key=os.getenv("OPENAI_API_KEY"), **kw
    ),
    vision_model_func=vision_model_func,
)

# Pre-parsed content from external OCR

content = [
    {"type": "text", "text": "RAGAnything processing pipeline.", "page_idx": 0},
    {"type": "image", "img_path": "chart.png", "image_caption": ["Revenue Chart"], "page_idx": 1},
]

async def main():
    await rag.insert_content_list(content, doc_id="ocr-batch-001")
    answer = await rag.aquery("What does the chart show?", mode="hybrid")
    logger.info(f"Answer: {answer}")

if __name__ == "__main__":
    asyncio.run(main())

Configuration Options and Callbacks

The insertion behavior is controlled via RAGAnythingConfig and optional callback hooks:

• display_content_stats – Boolean flag to emit processing statistics automatically.
• Callback hooks – Implement on_text_insert_start, on_text_insert_complete, and on_document_complete in a CallbackManager to monitor insertion progress.
• Context window – The ContextExtractor in raganything/enhanced_markdown.py uses configurable window sizes to gather surrounding text for multimodal captioning.

When doc_id is omitted, the system generates a deterministic hash based on the content list content, ensuring idempotent inserts for identical content while preventing collisions for distinct data.

Summary

• Bypass parsing by using insert_content_list from ProcessorMixin to feed structured data directly into HKUDS/RAG-Anything.
• Structure content as a list of typed dictionaries (text, image, table, equation) with page_idx metadata for optimal context extraction.
• Automatic handling includes LightRAG initialization (raganything/raganything.py), content separation, multimodal processing via dedicated processors (raganything/modalprocessors.py), and knowledge graph construction.
• Context awareness is preserved through the ContextExtractor, allowing multimodal items to reference surrounding text during caption generation.
• Callbacks and statistics provide observability into the insertion pipeline without requiring parser execution.

Frequently Asked Questions

What format should the pre-parsed content list follow?

The content list must be a Python list containing dictionaries. Each dictionary requires a type key (one of text, image, table, equation) and a page_idx integer. Images require img_path, image_caption, and image_footnote keys. Tables require table_body, table_caption, and table_footnote. Equations require latex and text fields. Text blocks require only a text string. This structure mirrors the output of RAGAnything's internal parsers, ensuring compatibility with downstream processing.

How does RAGAnything handle document IDs when inserting content lists?

If you omit the doc_id parameter, the system automatically generates a deterministic hash using _generate_content_based_doc_id (line 176–179 in processor.py). This hash is derived from the content list itself, ensuring that identical content produces the same identifier for idempotent operations. You can override this by providing your own doc_id string, which is useful for tracking external document identifiers or implementing custom versioning schemes.

Can I insert mixed content types (text and images) in a single call?

Yes. The insert_content_list method is designed for heterogeneous content. The internal separate_content function splits your list into text and multimodal streams. Text is chunked and embedded immediately, while images, tables, and equations are routed to their respective processors (ImageModalProcessor, TableModalProcessor, EquationModalProcessor). The system maintains cross-references between these modalities through the context extractor, ensuring multimodal queries can retrieve relevant text and visual content simultaneously.

What happens if LightRAG hasn't been initialized when I call insert_content_list?

The RAGAnything class automatically initializes the LightRAG backend before processing begins. The method _ensure_lightrag_initialized() (lines 55–61 in raganything/raganything.py) checks for existing storage instances and creates them if missing, using your configured working_dir and embedding functions. This guarantees that vector stores and knowledge graph databases are ready to receive content regardless of insertion order or timing in your application lifecycle.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}