How to Enable VLM-Enhanced Queries for Image Analysis in RAGAnything

Supply a vision_model_func when initializing RAGAnything to automatically enable multimodal processing, then call aquery() with vlm_enhanced=True (or omit the flag) to analyze images retrieved during retrieval-augmented generation.

RAGAnything extends LightRAG with native vision-language model (VLM) support, allowing you to query documents containing images using multimodal AI. By providing a compatible vision model function during initialization, the system automatically routes appropriate queries through a VLM-enhanced pipeline that detects image references, encodes them to base64, and constructs multimodal message payloads for analysis.

Prerequisites for VLM-Enhanced Queries

To enable image analysis capabilities, you must provide a vision_model_func when constructing the RAGAnything object. According to the dataclass definition in raganything/raganything.py (lines 62-64), this optional attribute accepts a callable with the same signature as a standard LLM function but with additional support for image_data or full messages payloads.

If this attribute is present, RAGAnything automatically sets vlm_enhanced=True by default, enabling seamless multimodal querying without additional configuration flags.

How VLM-Enhanced Querying Works

When you invoke aquery() or query(), the system evaluates whether to invoke the VLM pipeline based on the presence of your vision model function and the vlm_enhanced parameter.

Automatic Detection and Routing

The decision logic in raganything/query.py (lines 24-34) checks if vlm_enhanced is enabled and if self.vision_model_func exists. When both conditions are met, the query is automatically diverted to aquery_vlm_enhanced(). This method handles the complete multimodal pipeline:

1. Retrieves the baseline text-only retrieval prompt from LightRAG
2. Detects image paths in the prompt and replaces them with [VLM_IMAGE_n] markers
3. Encodes referenced images to base64 using safe validation
4. Builds the multimodal message payload aligned with text positions
5. Invokes the vision model via self.vision_model_func

Image Processing Pipeline

The core image handling occurs in two specialized methods within raganything/query.py. First, _process_image_paths_for_vlm() (lines 70-84) scans the retrieval prompt for image file paths, validates them using validate_image_file from utils.py, encodes them to base64 using encode_image_to_base64, and replaces the paths with indexed placeholders.

Next, _build_vlm_messages_with_images() (lines 92-106) constructs the final message payload by aligning the base64-encoded images with their corresponding text positions, creating a properly formatted multimodal input for your VLM.

Security and Safe Directories

The system restricts image file access to prevent accidental file-system reads. By default, image paths are limited to the working directory and parser output directory. You can expand these boundaries by passing the extra_safe_dirs argument to aquery_vlm_enhanced(), which accepts a list of additional directories to whitelist for image lookup.

Implementation Guide

Step 1: Define the Vision Model Function

Create a function that accepts multimodal inputs and forwards them to your VLM provider. The function must handle both pure-text fallback and full messages payload paths:

from lightrag.llm.openai import openai_complete_if_cache

def vision_model_func(
    prompt: str,
    system_prompt: str | None = None,
    history_messages: list = [],
    image_data: str | None = None,
    messages: list | None = None,
    **kwargs,
):
    # Multimodal path – VLM receives a full messages payload

    if messages:
        return openai_complete_if_cache(
            "gpt-4o", "", system_prompt=None, history_messages=[], 
            messages=messages, **kwargs
        )
    # Fallback – pure-text LLM

    return openai_complete_if_cache(
        "gpt-4o-mini", prompt, system_prompt, history_messages, **kwargs
    )

Step 2: Initialize RAGAnything with VLM Support

Pass your vision model function alongside your standard LLM function when creating the RAGAnything instance:

from raganything import RAGAnything, RAGAnythingConfig

config = RAGAnythingConfig(
    working_dir="./rag_storage",
    parser="mineru",
    enable_image_processing=True,
)

rag = RAGAnything(
    config=config,
    llm_model_func=lambda *a, **kw: openai_complete_if_cache("gpt-4o-mini", *a, **kw),
    vision_model_func=vision_model_func,
)

# Process a document first (required for retrieval)

await rag.process_document_complete("my_report.pdf", output_dir="./output")

Step 3: Execute VLM-Enhanced Queries

Once initialized, simply call aquery(). The system automatically uses VLM enhancement when vision_model_func is present:

# Automatic VLM enhancement

result = await rag.aquery(
    "What does the diagram on page 3 illustrate?",
    mode="hybrid"
)
print(result)

Or explicitly control the behavior:

# Force-disable VLM for text-only analysis

result = await rag.aquery(
    "Summarize the text on page 5.",
    mode="local",
    vlm_enhanced=False
)

Configuration Options

Disabling VLM for Specific Queries

To bypass image processing for particular queries where visual analysis is unnecessary, explicitly set vlm_enhanced=False in your aquery() call. This forces the system to use the standard text-only retrieval path even when a vision model is configured.

Expanding Safe Directories

When your images reside outside the default working directory, use aquery_vlm_enhanced() directly with the extra_safe_dirs parameter:

result = await rag.aquery_vlm_enhanced(
    "Explain the chart in ./extra_images/performance.png",
    mode="mix",
    extra_safe_dirs=["./extra_images"]
)

This configuration allows the pipeline to access images in ./extra_images/ while maintaining security restrictions against arbitrary file system access.

Summary

• Provide vision_model_func when initializing RAGAnything to enable automatic VLM enhancement (as defined in raganything/raganything.py lines 62-64)
• Call aquery() normally to trigger VLM-enhanced processing; the system detects image paths and routes to aquery_vlm_enhanced() automatically (logic in raganything/query.py lines 24-34)
• Images are encoded to base64 and replaced with markers via _process_image_paths_for_vlm() (lines 70-84) and _build_vlm_messages_with_images() (lines 92-106)
• Control access using extra_safe_dirs to whitelist additional image directories while maintaining security
• Override per-query using the vlm_enhanced boolean flag to force or suppress multimodal processing

Frequently Asked Questions

What is the exact signature required for vision_model_func?

The function must accept prompt, system_prompt, history_messages, image_data, and messages parameters, plus **kwargs. It should return the model response string. The implementation in raganything/raganything.py expects this interface to support both text-only fallback (when messages is None) and full multimodal vision processing (when messages contains the formatted payload).

Does enabling VLM support slow down text-only queries?

No. According to the routing logic in raganything/query.py (lines 24-34), text-only queries only incur VLM processing overhead when vlm_enhanced=True (explicitly or via auto-detection). You can maintain fast text queries by setting vlm_enhanced=False or omitting the vision_model_func entirely for text-only RAGAnything instances.

Which image formats are supported for VLM analysis?

The system validates images using validate_image_file from raganything/utils.py, which ensures only safe, supported formats are processed. While specific extensions depend on your underlying VLM provider, common formats like PNG, JPEG, and WebP are typically supported after passing the safety validation check.

Can I use different models for text generation and image analysis?

Yes. The llm_model_func and vision_model_func parameters are independent. You can configure a lightweight model like gpt-4o-mini for standard text retrieval and reasoning via llm_model_func, while using a more capable multimodal model like gpt-4o specifically for vision_model_func to handle image understanding tasks efficiently.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}