How to Configure Embedding and LLM Functions for OpenAI, vLLM, Ollama, and LMStudio in RAGAnything

RAGAnything configures embedding and LLM providers by accepting async Python functions that conform to the LightRAG API, allowing you to switch between OpenAI, Ollama, vLLM, and LMStudio without modifying core pipeline code.

RAGAnything decouples the retrieval pipeline from specific model providers through a function-based architecture. Instead of hard-coded API clients, the framework expects callable functions that match the LightRAG signatures for text generation and vector embedding. This design, implemented in raganything/raganything.py and configured via raganything/config.py, enables seamless provider swapping through environment variables or direct function injection.

Architecture Overview

RAGAnything uses a provider-agnostic pattern where async functions serve as adapters between the pipeline and any LLM or embedding service.

The initialization flow follows three layers:

1. Configuration Layer: RAGAnythingConfig (in raganything/config.py) reads environment variables for default settings but allows runtime overrides
2. Function Wrapping: The EmbeddingFunc class from lightrag/utils.py wraps raw embedding callables to make them pickle-safe and dimension-aware
3. Integration: RAGAnything.__post_init__ (in raganything/raganything.py) wires these functions into a LightRAG instance

Any async function returning strings for LLMs or List[List[float]] for embeddings satisfies the contract. This means you can connect OpenAI-compatible endpoints, local Ollama instances, or LM Studio servers by implementing the correct callables.

Step-by-Step Configuration Guide

Step 1: Create the LLM Callable

Define an async function matching the signature expected by LightRAG:

async def llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    # Implementation calls provider API

    return response_text

The function must accept prompt, optional system_prompt, history_messages, and return a plain string. According to raganything/raganything.py, this replaces the default model function during RAGAnything initialization.

Step 2: Create the Embedding Callable

Implement an async function that accepts a list of strings and returns a list of float vectors:

async def embedding_async(texts: List[str]) -> List[List[float]]:
    # Call provider embedding endpoint

    return embeddings  # List[List[float]]

Dimensionality must match your specific model (e.g., 768 for nomic-embed-text-v1.5, 1024 for bge-m3).

Step 3: Wrap with EmbeddingFunc

Import EmbeddingFunc from lightrag.utils and wrap your embedding callable:

from lightrag.utils import EmbeddingFunc

embedding_func = EmbeddingFunc(
    embedding_dim=768,        # Must match your model's output dimension

    max_token_size=8192,      # Max tokens per embedding request

    func=embedding_async,     # Your async function from Step 2

)

This wrapper ensures compatibility with LightRAG's caching mechanism and makes the function pickle-safe for multiprocessing.

Step 4: Initialize RAGAnything

Pass both functions to the RAGAnything constructor alongside your configuration:

from raganything import RAGAnything, RAGAnythingConfig

config = RAGAnythingConfig(working_dir="./rag_storage")

rag = RAGAnything(
    config=config,
    llm_model_func=llm_model_func,      # From Step 1

    embedding_func=embedding_func,      # From Step 3

)

Provider-Specific Implementations

OpenAI and vLLM

For OpenAI-compatible endpoints (including vLLM servers), use the helper functions from LightRAG's OpenAI module.

In examples/vllm_integration_example.py, the implementation uses openai_complete_if_cache for LLM calls:

from lightrag.llm.openai import openai_complete_if_cache, openai_embed

async def vllm_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    return await openai_complete_if_cache(
        model=VLLM_MODEL_NAME,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=VLLM_BASE_URL,          # Your vLLM endpoint

        api_key=VLLM_API_KEY,
        **kwargs,
    )

For embeddings in the same file:

async def vllm_embedding_async(texts: List[str]) -> List[List[float]]:
    embeddings = await openai_embed(
        texts=texts,
        model=VLLM_EMBED_MODEL,
        base_url=VLLM_EMBED_BASE_URL,
        api_key=VLLM_EMBED_API_KEY,
    )
    return embeddings.tolist()

Wrap with EmbeddingFunc(embedding_dim=1024, ...) for models like bge-m3.

Ollama

For local Ollama instances, use the official ollama Python client. As shown in examples/ollama_integration_example.py:

import ollama

async def ollama_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    # Ollama exposes an OpenAI-compatible endpoint

    return await openai_complete_if_cache(
        model=OLLAMA_LLM_MODEL,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=OLLAMA_BASE_URL,
        api_key=OLLAMA_API_KEY,
        **kwargs,
    )

async def ollama_embedding_async(texts: List[str]) -> List[List[float]]:
    client = ollama.AsyncClient(host=OLLAMA_HOST)
    response = await client.embed(
        model=OLLAMA_EMBEDDING_MODEL,
        input=texts
    )
    return response.embeddings

Configure the wrapper with the correct dimension (e.g., 768 for nomic-embed-text):

EmbeddingFunc(
    embedding_dim=768,
    max_token_size=8192,
    func=ollama_embedding_async,
)

LM Studio

LM Studio runs local OpenAI-compatible servers. The pattern mirrors OpenAI but targets your local endpoint, as demonstrated in examples/lmstudio_integration_example.py:

async def lmstudio_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    return await openai_complete_if_cache(
        model=LM_MODEL_NAME,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=LM_BASE_URL,        # Usually http://localhost:1234/v1

        api_key=LM_API_KEY,
        **kwargs,
    )

async def lmstudio_embedding_async(texts: List[str]) -> List[List[float]]:
    embeddings = await openai_embed(
        texts=texts,
        model=LM_EMBED_MODEL,
        base_url=LM_BASE_URL,
        api_key=LM_API_KEY,
    )
    return embeddings.tolist()

For nomic-embed-text-v1.5, use EmbeddingFunc(embedding_dim=768, ...).

Complete Integration Example

This example demonstrates wiring an Ollama backend with multimodal processing enabled:

import asyncio
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.utils import EmbeddingFunc

# Configuration

config = RAGAnythingConfig(
    working_dir="./rag_storage",
    parser="mineru",
    enable_table_processing=True,
    enable_equation_processing=True,
)

# Provider functions (Ollama example)

async def my_llm_func(prompt, system_prompt=None, history_messages=None, **kwargs):
    import ollama
    # Simplified example - see full implementation above

    pass

async def my_embed_func(texts):
    import ollama
    client = ollama.AsyncClient(host="http://localhost:11434")
    response = await client.embed(model="nomic-embed-text", input=texts)
    return response.embeddings

# Initialize

rag = RAGAnything(
    config=config,
    llm_model_func=my_llm_func,
    embedding_func=EmbeddingFunc(
        embedding_dim=768,
        max_token_size=8192,
        func=my_embed_func
    ),
)

# Usage

async def main():
    await rag.process_document_complete(
        file_path="document.pdf",
        output_dir="./output",
        parse_method="auto"
    )
    result = await rag.aquery("What are the key findings?", mode="hybrid")
    print(result)

asyncio.run(main())

Summary

• RAGAnything expects async functions, not hard-coded clients, for both LLM and embedding operations
• Wrap embedding functions with EmbeddingFunc from lightrag/utils.py, specifying the correct embedding_dim for your model
• OpenAI-compatible endpoints (including vLLM and LM Studio) use openai_complete_if_cache and openai_embed from LightRAG
• Ollama requires the ollama Python client for embeddings while reusing OpenAI-compatible endpoints for LLM chat
• Provider switching requires only changing the function implementations or environment variables in RAGAnythingConfig, with no changes to raganything/raganything.py

Frequently Asked Questions

Can I use a custom local model not listed in the examples?

Yes. Any model exposing an OpenAI-compatible REST API (chat completions and embeddings) works with the openai_complete_if_cache and openai_embed helpers. For proprietary protocols, implement the async function signatures described in Step 1 and Step 2, returning strings for LLM responses and List[List[float]] for embeddings.

What embedding dimension should I specify in EmbeddingFunc?

Specify the output dimension of your specific model. Common values include 768 for nomic-embed-text-v1.5 and Ollama's Nomic embeddings, 1024 for bge-m3, and 1536 for OpenAI's text-embedding-3-small. Mismatched dimensions will cause vector database errors during initialization in raganything/raganything.py.

Do I need to modify core RAGAnything code to add a new provider?

No. Create your provider-specific async functions in your application code or a separate module, then pass them to the RAGAnything constructor. The core library in raganything/raganything.py treats these as black-box callables, so no source modifications are necessary.

How does function caching work across different providers?

LightRAG automatically caches LLM and embedding calls based on function arguments and input text hashes. The EmbeddingFunc wrapper in lightrag/utils.py ensures your custom functions are pickle-safe for this caching mechanism. Cache storage locations are controlled via RAGAnythingConfig environment variables.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}