# How to Configure Embedding and LLM Functions for OpenAI, vLLM, Ollama, and LMStudio in RAGAnything

> Learn to configure embedding and LLM functions for OpenAI, vLLM, Ollama, and LMStudio in RAGAnything. Streamline your RAG pipeline with flexible provider integration.

- Repository: [✨Data Intelligence Lab@HKU✨/RAG-Anything](https://github.com/HKUDS/RAG-Anything)
- Tags: how-to-guide
- Published: 2026-04-22

---

**RAGAnything configures embedding and LLM providers by accepting async Python functions that conform to the LightRAG API, allowing you to switch between OpenAI, Ollama, vLLM, and LMStudio without modifying core pipeline code.**

RAGAnything decouples the retrieval pipeline from specific model providers through a function-based architecture. Instead of hard-coded API clients, the framework expects callable functions that match the LightRAG signatures for text generation and vector embedding. This design, implemented in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py) and configured via [`raganything/config.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/config.py), enables seamless provider swapping through environment variables or direct function injection.

## Architecture Overview

RAGAnything uses a provider-agnostic pattern where **async functions** serve as adapters between the pipeline and any LLM or embedding service.

The initialization flow follows three layers:

1. **Configuration Layer**: `RAGAnythingConfig` (in [`raganything/config.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/config.py)) reads environment variables for default settings but allows runtime overrides
2. **Function Wrapping**: The `EmbeddingFunc` class from [`lightrag/utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/lightrag/utils.py) wraps raw embedding callables to make them pickle-safe and dimension-aware
3. **Integration**: `RAGAnything.__post_init__` (in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py)) wires these functions into a `LightRAG` instance

Any async function returning strings for LLMs or `List[List[float]]` for embeddings satisfies the contract. This means you can connect OpenAI-compatible endpoints, local Ollama instances, or LM Studio servers by implementing the correct callables.

## Step-by-Step Configuration Guide

### Step 1: Create the LLM Callable

Define an async function matching the signature expected by LightRAG:

```python
async def llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    # Implementation calls provider API

    return response_text

```

The function must accept `prompt`, optional `system_prompt`, `history_messages`, and return a plain string. According to [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py), this replaces the default model function during `RAGAnything` initialization.

### Step 2: Create the Embedding Callable

Implement an async function that accepts a list of strings and returns a list of float vectors:

```python
async def embedding_async(texts: List[str]) -> List[List[float]]:
    # Call provider embedding endpoint

    return embeddings  # List[List[float]]

```

Dimensionality must match your specific model (e.g., **768** for `nomic-embed-text-v1.5`, **1024** for `bge-m3`).

### Step 3: Wrap with EmbeddingFunc

Import `EmbeddingFunc` from `lightrag.utils` and wrap your embedding callable:

```python
from lightrag.utils import EmbeddingFunc

embedding_func = EmbeddingFunc(
    embedding_dim=768,        # Must match your model's output dimension

    max_token_size=8192,      # Max tokens per embedding request

    func=embedding_async,     # Your async function from Step 2

)

```

This wrapper ensures compatibility with LightRAG's caching mechanism and makes the function pickle-safe for multiprocessing.

### Step 4: Initialize RAGAnything

Pass both functions to the `RAGAnything` constructor alongside your configuration:

```python
from raganything import RAGAnything, RAGAnythingConfig

config = RAGAnythingConfig(working_dir="./rag_storage")

rag = RAGAnything(
    config=config,
    llm_model_func=llm_model_func,      # From Step 1

    embedding_func=embedding_func,      # From Step 3

)

```

## Provider-Specific Implementations

### OpenAI and vLLM

For OpenAI-compatible endpoints (including vLLM servers), use the helper functions from LightRAG's OpenAI module.

In [`examples/vllm_integration_example.py`](https://github.com/HKUDS/RAG-Anything/blob/main/examples/vllm_integration_example.py), the implementation uses `openai_complete_if_cache` for LLM calls:

```python
from lightrag.llm.openai import openai_complete_if_cache, openai_embed

async def vllm_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    return await openai_complete_if_cache(
        model=VLLM_MODEL_NAME,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=VLLM_BASE_URL,          # Your vLLM endpoint

        api_key=VLLM_API_KEY,
        **kwargs,
    )

```

For embeddings in the same file:

```python
async def vllm_embedding_async(texts: List[str]) -> List[List[float]]:
    embeddings = await openai_embed(
        texts=texts,
        model=VLLM_EMBED_MODEL,
        base_url=VLLM_EMBED_BASE_URL,
        api_key=VLLM_EMBED_API_KEY,
    )
    return embeddings.tolist()

```

Wrap with `EmbeddingFunc(embedding_dim=1024, ...)` for models like `bge-m3`.

### Ollama

For local Ollama instances, use the official `ollama` Python client. As shown in [`examples/ollama_integration_example.py`](https://github.com/HKUDS/RAG-Anything/blob/main/examples/ollama_integration_example.py):

```python
import ollama

async def ollama_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    # Ollama exposes an OpenAI-compatible endpoint

    return await openai_complete_if_cache(
        model=OLLAMA_LLM_MODEL,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=OLLAMA_BASE_URL,
        api_key=OLLAMA_API_KEY,
        **kwargs,
    )

async def ollama_embedding_async(texts: List[str]) -> List[List[float]]:
    client = ollama.AsyncClient(host=OLLAMA_HOST)
    response = await client.embed(
        model=OLLAMA_EMBEDDING_MODEL,
        input=texts
    )
    return response.embeddings

```

Configure the wrapper with the correct dimension (e.g., **768** for `nomic-embed-text`):

```python
EmbeddingFunc(
    embedding_dim=768,
    max_token_size=8192,
    func=ollama_embedding_async,
)

```

### LM Studio

LM Studio runs local OpenAI-compatible servers. The pattern mirrors OpenAI but targets your local endpoint, as demonstrated in [`examples/lmstudio_integration_example.py`](https://github.com/HKUDS/RAG-Anything/blob/main/examples/lmstudio_integration_example.py):

```python
async def lmstudio_llm_model_func(
    prompt: str,
    system_prompt: Optional[str] = None,
    history_messages: List[Dict] = None,
    **kwargs,
) -> str:
    return await openai_complete_if_cache(
        model=LM_MODEL_NAME,
        prompt=prompt,
        system_prompt=system_prompt,
        history_messages=history_messages or [],
        base_url=LM_BASE_URL,        # Usually http://localhost:1234/v1

        api_key=LM_API_KEY,
        **kwargs,
    )

async def lmstudio_embedding_async(texts: List[str]) -> List[List[float]]:
    embeddings = await openai_embed(
        texts=texts,
        model=LM_EMBED_MODEL,
        base_url=LM_BASE_URL,
        api_key=LM_API_KEY,
    )
    return embeddings.tolist()

```

For `nomic-embed-text-v1.5`, use `EmbeddingFunc(embedding_dim=768, ...)`.

## Complete Integration Example

This example demonstrates wiring an Ollama backend with multimodal processing enabled:

```python
import asyncio
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.utils import EmbeddingFunc

# Configuration

config = RAGAnythingConfig(
    working_dir="./rag_storage",
    parser="mineru",
    enable_table_processing=True,
    enable_equation_processing=True,
)

# Provider functions (Ollama example)

async def my_llm_func(prompt, system_prompt=None, history_messages=None, **kwargs):
    import ollama
    # Simplified example - see full implementation above

    pass

async def my_embed_func(texts):
    import ollama
    client = ollama.AsyncClient(host="http://localhost:11434")
    response = await client.embed(model="nomic-embed-text", input=texts)
    return response.embeddings

# Initialize

rag = RAGAnything(
    config=config,
    llm_model_func=my_llm_func,
    embedding_func=EmbeddingFunc(
        embedding_dim=768,
        max_token_size=8192,
        func=my_embed_func
    ),
)

# Usage

async def main():
    await rag.process_document_complete(
        file_path="document.pdf",
        output_dir="./output",
        parse_method="auto"
    )
    result = await rag.aquery("What are the key findings?", mode="hybrid")
    print(result)

asyncio.run(main())

```

## Summary

- **RAGAnything expects async functions**, not hard-coded clients, for both LLM and embedding operations
- **Wrap embedding functions** with `EmbeddingFunc` from [`lightrag/utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/lightrag/utils.py), specifying the correct `embedding_dim` for your model
- **OpenAI-compatible endpoints** (including vLLM and LM Studio) use `openai_complete_if_cache` and `openai_embed` from LightRAG
- **Ollama** requires the `ollama` Python client for embeddings while reusing OpenAI-compatible endpoints for LLM chat
- **Provider switching** requires only changing the function implementations or environment variables in `RAGAnythingConfig`, with no changes to [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py)

## Frequently Asked Questions

### Can I use a custom local model not listed in the examples?

Yes. Any model exposing an OpenAI-compatible REST API (chat completions and embeddings) works with the `openai_complete_if_cache` and `openai_embed` helpers. For proprietary protocols, implement the async function signatures described in Step 1 and Step 2, returning strings for LLM responses and `List[List[float]]` for embeddings.

### What embedding dimension should I specify in EmbeddingFunc?

Specify the output dimension of your specific model. Common values include **768** for `nomic-embed-text-v1.5` and Ollama's Nomic embeddings, **1024** for `bge-m3`, and **1536** for OpenAI's `text-embedding-3-small`. Mismatched dimensions will cause vector database errors during initialization in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py).

### Do I need to modify core RAGAnything code to add a new provider?

No. Create your provider-specific async functions in your application code or a separate module, then pass them to the `RAGAnything` constructor. The core library in [`raganything/raganything.py`](https://github.com/HKUDS/RAG-Anything/blob/main/raganything/raganything.py) treats these as black-box callables, so no source modifications are necessary.

### How does function caching work across different providers?

LightRAG automatically caches LLM and embedding calls based on function arguments and input text hashes. The `EmbeddingFunc` wrapper in [`lightrag/utils.py`](https://github.com/HKUDS/RAG-Anything/blob/main/lightrag/utils.py) ensures your custom functions are pickle-safe for this caching mechanism. Cache storage locations are controlled via `RAGAnythingConfig` environment variables.