# How to Enable VLM-Enhanced Queries for Image Analysis in RAGAnything

> Learn to enable VLM-enhanced queries for image analysis in RAGAnything. Configure multimodal processing and analyze images with aquery() calls.

- Repository: [✨Data Intelligence Lab@HKU✨/RAG-Anything](https://github.com/HKUDS/RAG-Anything)
- Tags: how-to-guide
- Published: 2026-04-22

---

**Supply a `vision_model_func` when initializing `RAGAnything` to automatically enable multimodal processing, then call `aquery()` with `vlm_enhanced=True` (or omit the flag) to analyze images retrieved during retrieval-augmented generation.**

RAGAnything extends LightRAG with native vision-language model (VLM) support, allowing you to query documents containing images using multimodal AI. By providing a compatible vision model function during initialization, the system automatically routes appropriate queries through a VLM-enhanced pipeline that detects image references, encodes them to base64, and constructs multimodal message payloads for analysis.

## Prerequisites for VLM-Enhanced Queries

To enable image analysis capabilities, you must provide a **`vision_model_func`** when constructing the `RAGAnything` object. According to the dataclass definition in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py) (lines 62-64), this optional attribute accepts a callable with the same signature as a standard LLM function but with additional support for `image_data` or full `messages` payloads.

If this attribute is present, `RAGAnything` automatically sets `vlm_enhanced=True` by default, enabling seamless multimodal querying without additional configuration flags.

## How VLM-Enhanced Querying Works

When you invoke `aquery()` or `query()`, the system evaluates whether to invoke the VLM pipeline based on the presence of your vision model function and the `vlm_enhanced` parameter.

### Automatic Detection and Routing

The decision logic in [`raganything/query.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/query.py) (lines 24-34) checks if `vlm_enhanced` is enabled and if `self.vision_model_func` exists. When both conditions are met, the query is automatically diverted to `aquery_vlm_enhanced()`. This method handles the complete multimodal pipeline:

1. Retrieves the baseline text-only retrieval prompt from LightRAG
2. Detects image paths in the prompt and replaces them with `[VLM_IMAGE_n]` markers
3. Encodes referenced images to base64 using safe validation
4. Builds the multimodal message payload aligned with text positions
5. Invokes the vision model via `self.vision_model_func`

### Image Processing Pipeline

The core image handling occurs in two specialized methods within [`raganything/query.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/query.py). First, `_process_image_paths_for_vlm()` (lines 70-84) scans the retrieval prompt for image file paths, validates them using `validate_image_file` from [`utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/utils.py), encodes them to base64 using `encode_image_to_base64`, and replaces the paths with indexed placeholders.

Next, `_build_vlm_messages_with_images()` (lines 92-106) constructs the final message payload by aligning the base64-encoded images with their corresponding text positions, creating a properly formatted multimodal input for your VLM.

### Security and Safe Directories

The system restricts image file access to prevent accidental file-system reads. By default, image paths are limited to the working directory and parser output directory. You can expand these boundaries by passing the **`extra_safe_dirs`** argument to `aquery_vlm_enhanced()`, which accepts a list of additional directories to whitelist for image lookup.

## Implementation Guide

### Step 1: Define the Vision Model Function

Create a function that accepts multimodal inputs and forwards them to your VLM provider. The function must handle both pure-text fallback and full `messages` payload paths:

```python
from lightrag.llm.openai import openai_complete_if_cache

def vision_model_func(
    prompt: str,
    system_prompt: str | None = None,
    history_messages: list = [],
    image_data: str | None = None,
    messages: list | None = None,
    **kwargs,
):
    # Multimodal path – VLM receives a full messages payload

    if messages:
        return openai_complete_if_cache(
            "gpt-4o", "", system_prompt=None, history_messages=[], 
            messages=messages, **kwargs
        )
    # Fallback – pure-text LLM

    return openai_complete_if_cache(
        "gpt-4o-mini", prompt, system_prompt, history_messages, **kwargs
    )

```

### Step 2: Initialize RAGAnything with VLM Support

Pass your vision model function alongside your standard LLM function when creating the `RAGAnything` instance:

```python
from raganything import RAGAnything, RAGAnythingConfig

config = RAGAnythingConfig(
    working_dir="./rag_storage",
    parser="mineru",
    enable_image_processing=True,
)

rag = RAGAnything(
    config=config,
    llm_model_func=lambda *a, **kw: openai_complete_if_cache("gpt-4o-mini", *a, **kw),
    vision_model_func=vision_model_func,
)

# Process a document first (required for retrieval)

await rag.process_document_complete("my_report.pdf", output_dir="./output")

```

### Step 3: Execute VLM-Enhanced Queries

Once initialized, simply call `aquery()`. The system automatically uses VLM enhancement when `vision_model_func` is present:

```python

# Automatic VLM enhancement

result = await rag.aquery(
    "What does the diagram on page 3 illustrate?",
    mode="hybrid"
)
print(result)

```

Or explicitly control the behavior:

```python

# Force-disable VLM for text-only analysis

result = await rag.aquery(
    "Summarize the text on page 5.",
    mode="local",
    vlm_enhanced=False
)

```

## Configuration Options

### Disabling VLM for Specific Queries

To bypass image processing for particular queries where visual analysis is unnecessary, explicitly set **`vlm_enhanced=False`** in your `aquery()` call. This forces the system to use the standard text-only retrieval path even when a vision model is configured.

### Expanding Safe Directories

When your images reside outside the default working directory, use `aquery_vlm_enhanced()` directly with the `extra_safe_dirs` parameter:

```python
result = await rag.aquery_vlm_enhanced(
    "Explain the chart in ./extra_images/performance.png",
    mode="mix",
    extra_safe_dirs=["./extra_images"]
)

```

This configuration allows the pipeline to access images in `./extra_images/` while maintaining security restrictions against arbitrary file system access.

## Summary

- **Provide `vision_model_func`** when initializing `RAGAnything` to enable automatic VLM enhancement (as defined in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py) lines 62-64)
- **Call `aquery()`** normally to trigger VLM-enhanced processing; the system detects image paths and routes to `aquery_vlm_enhanced()` automatically (logic in [`raganything/query.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/query.py) lines 24-34)
- **Images are encoded to base64** and replaced with markers via `_process_image_paths_for_vlm()` (lines 70-84) and `_build_vlm_messages_with_images()` (lines 92-106)
- **Control access** using `extra_safe_dirs` to whitelist additional image directories while maintaining security
- **Override per-query** using the `vlm_enhanced` boolean flag to force or suppress multimodal processing

## Frequently Asked Questions

### What is the exact signature required for `vision_model_func`?

The function must accept `prompt`, `system_prompt`, `history_messages`, `image_data`, and `messages` parameters, plus `**kwargs`. It should return the model response string. The implementation in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py) expects this interface to support both text-only fallback (when `messages` is None) and full multimodal vision processing (when `messages` contains the formatted payload).

### Does enabling VLM support slow down text-only queries?

No. According to the routing logic in [`raganything/query.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/query.py) (lines 24-34), text-only queries only incur VLM processing overhead when `vlm_enhanced=True` (explicitly or via auto-detection). You can maintain fast text queries by setting `vlm_enhanced=False` or omitting the `vision_model_func` entirely for text-only RAGAnything instances.

### Which image formats are supported for VLM analysis?

The system validates images using `validate_image_file` from [`raganything/utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/utils.py), which ensures only safe, supported formats are processed. While specific extensions depend on your underlying VLM provider, common formats like PNG, JPEG, and WebP are typically supported after passing the safety validation check.

### Can I use different models for text generation and image analysis?

Yes. The `llm_model_func` and `vision_model_func` parameters are independent. You can configure a lightweight model like `gpt-4o-mini` for standard text retrieval and reasoning via `llm_model_func`, while using a more capable multimodal model like `gpt-4o` specifically for `vision_model_func` to handle image understanding tasks efficiently.