# How to Set Up LightRAG with Ollama for Local LLM Inference and Context Size Configuration

> Learn to set up LightRAG with Ollama for local LLM inference. Configure context size and optimize your local language model performance with this easy guide.

- Repository: [✨Data Intelligence Lab@HKU✨/LightRAG](https://github.com/HKUDS/LightRAG)
- Tags: how-to-guide
- Published: 2026-03-23

---

**LightRAG integrates with Ollama through the `ollama_model_complete` wrapper in [`lightrag/llm/ollama.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/llm/ollama.py), enabling local inference with configurable context windows by passing `options={"num_ctx": 8192}` inside `llm_model_kwargs` when instantiating the `LightRAG` class.**

LightRAG provides a modular retrieval-augmented generation architecture that decouples the core orchestrator from specific LLM providers. When you set up LightRAG with Ollama, the framework binds to your local Ollama server via async HTTP bindings, allowing you to control the context window size through the `num_ctx` parameter and handle both chat completions and embeddings through a unified interface.

## Understanding the Ollama Architecture in LightRAG

The Ollama integration resides in [`lightrag/llm/ollama.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/llm/ollama.py) and consists of thin async wrappers around Ollama's HTTP API. This design separates the **LLM layer**, **embedding layer**, and **storage adapters**, making the provider interchangeable without modifying core RAG logic.

### Model Completion Wrapper

The `ollama_model_complete` function serves as the primary bridge between LightRAG and Ollama. It retrieves the model name from the global configuration via `hashing_kv.global_config["llm_model_name"]` and forwards prompts to the internal `_ollama_model_if_cache` helper (lines 53-73). This wrapper handles both streaming and non-streaming responses while ensuring all network resources close properly in `finally` blocks.

### Host Selection Logic

LightRAG automatically selects between local and cloud endpoints through the private `_coerce_host_for_cloud_model` helper (lines 38-51). If your model name ends with `-cloud` or `:cloud`, the function switches the host from `http://localhost:11434` to the Ollama Cloud endpoint, enabling seamless hybrid deployments without changing application code.

### Retry and Streaming Configuration

All Ollama requests implement a Tenacity retry policy with a maximum of three attempts and exponential backoff (lines 61-84 and 95-112). The wrapper supports streaming responses for real-time token generation and automatically retries on transient network errors, ensuring robust local inference even under resource constraints.

## Configuring Local LLM Inference

To instantiate LightRAG with Ollama, you pass the `ollama_model_complete` function to the `llm_model_func` parameter along with connection details in `llm_model_kwargs`:

```python
from lightrag import LightRAG
from lightrag.llm.ollama import ollama_model_complete

rag = LightRAG(
    working_dir="./rag_workspace",
    llm_model_func=ollama_model_complete,
    llm_model_name="qwen2.5-coder:7b",
    llm_model_kwargs={
        "host": "http://localhost:11434",
        "options": {"num_ctx": 8192},
        "timeout": 300,
    },
)

```

The `host` parameter defaults to `http://localhost:11434` for local Ollama servers. The `timeout` value (in seconds) prevents hanging requests during large context processing.

## Managing Context Size with num_ctx

LightRAG does not enforce hard token limits internally; instead, it relies on Ollama's `num_ctx` configuration to define the context window. You specify this through the `options` dictionary within `llm_model_kwargs` (as demonstrated in [`examples/lightrag_ollama_demo.py`](https://github.com/HKUDS/LightRAG/blob/main/examples/lightrag_ollama_demo.py), lines 90-94):

```python
llm_model_kwargs={
    "host": "http://localhost:11434",
    "options": {"num_ctx": 8192},  # Allocates 8K token window

    "timeout": 300,
}

```

When you set `num_ctx` to `8192`, Ollama allocates an 8,000-token context window for every chat request. The framework forwards this option directly to Ollama's API, allowing you to match the context size to your specific model's capabilities (e.g., 4K, 8K, or 128K contexts) without modifying LightRAG's core code.

## Setting Up Ollama Embeddings

LightRAG handles embeddings through the `ollama_embed` function, decorated with `@wrap_embedding_func_with_attrs` to expose the model's dimensionality (`1024`) and maximum token window (`8192`). The implementation resides at lines 75-84 and 90-97 in [`lightrag/llm/ollama.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/llm/ollama.py).

Unlike the LLM wrapper, the embedding function does not slice texts client-side. Instead, it forwards the complete list of texts to `ollama.AsyncClient.embed`, allowing Ollama to handle truncation according to its own `num_ctx` settings:

```python
from functools import partial
from lightrag.utils import EmbeddingFunc
from lightrag.llm.ollama import ollama_embed

embedding_func = EmbeddingFunc(
    embedding_dim=1024,
    max_token_size=8192,
    func=partial(
        ollama_embed.func,
        embed_model="bge-m3:latest",
        host="http://localhost:11434",
    ),
)

```

Pass this `embedding_func` to the `embedding_func` parameter when initializing `LightRAG`.

## Complete Implementation Example

The following production-ready script demonstrates the full integration, including environment variable configuration, context size management, and graceful shutdown:

```python
import os
import asyncio
from functools import partial

from lightrag import LightRAG, QueryParam
from lightrag.llm.ollama import ollama_model_complete, ollama_embed
from lightrag.utils import EmbeddingFunc, set_verbose_debug

# Configuration via environment variables

os.environ.setdefault("LLM_MODEL", "qwen2.5-coder:7b")
os.environ.setdefault("LLM_BINDING_HOST", "http://localhost:11434")
os.environ.setdefault("EMBEDDING_MODEL", "bge-m3:latest")
os.environ.setdefault("EMBEDDING_BINDING_HOST", "http://localhost:11434")
os.environ.setdefault("NUM_CTX", "8192")
os.environ.setdefault("TIMEOUT", "300")

async def build_rag():
    # Initialize LightRAG with Ollama bindings

    rag = LightRAG(
        working_dir="./rag_workspace",
        llm_model_func=ollama_model_complete,
        llm_model_name=os.getenv("LLM_MODEL"),
        llm_model_kwargs={
            "host": os.getenv("LLM_BINDING_HOST"),
            "options": {"num_ctx": int(os.getenv("NUM_CTX"))},
            "timeout": int(os.getenv("TIMEOUT")),
        },
        embedding_func=EmbeddingFunc(
            embedding_dim=1024,
            max_token_size=8192,
            func=partial(
                ollama_embed.func,
                embed_model=os.getenv("EMBEDDING_MODEL"),
                host=os.getenv("EMBEDDING_BINDING_HOST"),
            ),
        ),
    )
    await rag.initialize_storages()

    # Insert documents

    await rag.ainsert("Your document text goes here...")

    # Query with streaming

    resp = await rag.aquery(
        "Summarize the main ideas.",
        param=QueryParam(mode="hybrid", stream=True),
    )
    async for chunk in resp:
        print(chunk, end="", flush=True)

    # Cleanup

    await rag.finalize_storages()

if __name__ == "__main__":
    set_verbose_debug(True)
    asyncio.run(build_rag())

```

This implementation references the official [`lightrag_ollama_demo.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag_ollama_demo.py) example and demonstrates how to wire the `ollama_model_complete` and `ollama_embed` functions into the core `LightRAG` orchestrator.

## Summary

- **Architecture**: LightRAG's Ollama integration ([`lightrag/llm/ollama.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/llm/ollama.py)) provides async wrappers that handle host selection, retries, and resource cleanup automatically.
- **Context Control**: Pass `options={"num_ctx": 8192}` in `llm_model_kwargs` to configure Ollama's context window; LightRAG defers token limit enforcement to the underlying Ollama model.
- **Host Flexibility**: The `_coerce_host_for_cloud_model` helper switches to Ollama Cloud automatically when model names end with `-cloud` or `:cloud`.
- **Embeddings**: The `ollama_embed` function exposes 1024-dimensional vectors with an 8192-token max window, delegating truncation to Ollama's server-side logic.
- **Resilience**: Built-in Tenacity retry logic (3 attempts, exponential backoff) ensures stable local inference even during resource contention.

## Frequently Asked Questions

### How do I increase the context window beyond 8K tokens when using LightRAG with Ollama?

Increase the `num_ctx` value in the `options` dictionary within `llm_model_kwargs`. For example, set `options={"num_ctx": 32768}` for a 32K context window. LightRAG forwards this parameter directly to Ollama's API without enforcing upper limits, though you must ensure your local model (e.g., Llama 3.1, Qwen2.5) actually supports the requested context length.

### Can I switch between local Ollama and Ollama Cloud without changing my code?

Yes. The Ollama binding automatically detects cloud models through the `_coerce_host_for_cloud_model` function. If your `llm_model_name` ends with `-cloud` or `:cloud`, LightRAG switches the host from `localhost:11434` to the Ollama Cloud endpoint. Otherwise, it defaults to the local host specified in your configuration.

### Why does LightRAG not truncate my texts when using Ollama embeddings?

The `ollama_embed` function (lines 75-84 in [`lightrag/llm/ollama.py`](https://github.com/HKUDS/LightRAG/blob/main/lightrag/llm/ollama.py)) forwards the complete text list to `ollama.AsyncClient.embed` and does not implement client-side truncation. Ollama handles truncation internally based on its own `num_ctx` settings. If you require strict token limits, configure `num_ctx` in your Ollama server configuration or pre-process texts before insertion.

### How does LightRAG handle connection timeouts with local Ollama servers?

The `ollama_model_complete` function accepts a `timeout` parameter (in seconds) through `llm_model_kwargs`. The implementation uses Tenacity to retry failed requests up to three times with exponential backoff (lines 61-84). Set `timeout` to `300` or higher when processing large contexts to prevent premature connection drops during inference.