How to Set Up LightRAG with Ollama for Local LLM Inference and Context Size Configuration

LightRAG integrates with Ollama through the ollama_model_complete wrapper in lightrag/llm/ollama.py, enabling local inference with configurable context windows by passing options={"num_ctx": 8192} inside llm_model_kwargs when instantiating the LightRAG class.

LightRAG provides a modular retrieval-augmented generation architecture that decouples the core orchestrator from specific LLM providers. When you set up LightRAG with Ollama, the framework binds to your local Ollama server via async HTTP bindings, allowing you to control the context window size through the num_ctx parameter and handle both chat completions and embeddings through a unified interface.

Understanding the Ollama Architecture in LightRAG

The Ollama integration resides in lightrag/llm/ollama.py and consists of thin async wrappers around Ollama's HTTP API. This design separates the LLM layer, embedding layer, and storage adapters, making the provider interchangeable without modifying core RAG logic.

Model Completion Wrapper

The ollama_model_complete function serves as the primary bridge between LightRAG and Ollama. It retrieves the model name from the global configuration via hashing_kv.global_config["llm_model_name"] and forwards prompts to the internal _ollama_model_if_cache helper (lines 53-73). This wrapper handles both streaming and non-streaming responses while ensuring all network resources close properly in finally blocks.

Host Selection Logic

LightRAG automatically selects between local and cloud endpoints through the private _coerce_host_for_cloud_model helper (lines 38-51). If your model name ends with -cloud or :cloud, the function switches the host from http://localhost:11434 to the Ollama Cloud endpoint, enabling seamless hybrid deployments without changing application code.

Retry and Streaming Configuration

All Ollama requests implement a Tenacity retry policy with a maximum of three attempts and exponential backoff (lines 61-84 and 95-112). The wrapper supports streaming responses for real-time token generation and automatically retries on transient network errors, ensuring robust local inference even under resource constraints.

Configuring Local LLM Inference

To instantiate LightRAG with Ollama, you pass the ollama_model_complete function to the llm_model_func parameter along with connection details in llm_model_kwargs:

from lightrag import LightRAG
from lightrag.llm.ollama import ollama_model_complete

rag = LightRAG(
    working_dir="./rag_workspace",
    llm_model_func=ollama_model_complete,
    llm_model_name="qwen2.5-coder:7b",
    llm_model_kwargs={
        "host": "http://localhost:11434",
        "options": {"num_ctx": 8192},
        "timeout": 300,
    },
)

The host parameter defaults to http://localhost:11434 for local Ollama servers. The timeout value (in seconds) prevents hanging requests during large context processing.

Managing Context Size with num_ctx

LightRAG does not enforce hard token limits internally; instead, it relies on Ollama's num_ctx configuration to define the context window. You specify this through the options dictionary within llm_model_kwargs (as demonstrated in examples/lightrag_ollama_demo.py, lines 90-94):

llm_model_kwargs={
    "host": "http://localhost:11434",
    "options": {"num_ctx": 8192},  # Allocates 8K token window

    "timeout": 300,
}

When you set num_ctx to 8192, Ollama allocates an 8,000-token context window for every chat request. The framework forwards this option directly to Ollama's API, allowing you to match the context size to your specific model's capabilities (e.g., 4K, 8K, or 128K contexts) without modifying LightRAG's core code.

Setting Up Ollama Embeddings

LightRAG handles embeddings through the ollama_embed function, decorated with @wrap_embedding_func_with_attrs to expose the model's dimensionality (1024) and maximum token window (8192). The implementation resides at lines 75-84 and 90-97 in lightrag/llm/ollama.py.

Unlike the LLM wrapper, the embedding function does not slice texts client-side. Instead, it forwards the complete list of texts to ollama.AsyncClient.embed, allowing Ollama to handle truncation according to its own num_ctx settings:

from functools import partial
from lightrag.utils import EmbeddingFunc
from lightrag.llm.ollama import ollama_embed

embedding_func = EmbeddingFunc(
    embedding_dim=1024,
    max_token_size=8192,
    func=partial(
        ollama_embed.func,
        embed_model="bge-m3:latest",
        host="http://localhost:11434",
    ),
)

Pass this embedding_func to the embedding_func parameter when initializing LightRAG.

Complete Implementation Example

The following production-ready script demonstrates the full integration, including environment variable configuration, context size management, and graceful shutdown:

import os
import asyncio
from functools import partial

from lightrag import LightRAG, QueryParam
from lightrag.llm.ollama import ollama_model_complete, ollama_embed
from lightrag.utils import EmbeddingFunc, set_verbose_debug

# Configuration via environment variables

os.environ.setdefault("LLM_MODEL", "qwen2.5-coder:7b")
os.environ.setdefault("LLM_BINDING_HOST", "http://localhost:11434")
os.environ.setdefault("EMBEDDING_MODEL", "bge-m3:latest")
os.environ.setdefault("EMBEDDING_BINDING_HOST", "http://localhost:11434")
os.environ.setdefault("NUM_CTX", "8192")
os.environ.setdefault("TIMEOUT", "300")

async def build_rag():
    # Initialize LightRAG with Ollama bindings

    rag = LightRAG(
        working_dir="./rag_workspace",
        llm_model_func=ollama_model_complete,
        llm_model_name=os.getenv("LLM_MODEL"),
        llm_model_kwargs={
            "host": os.getenv("LLM_BINDING_HOST"),
            "options": {"num_ctx": int(os.getenv("NUM_CTX"))},
            "timeout": int(os.getenv("TIMEOUT")),
        },
        embedding_func=EmbeddingFunc(
            embedding_dim=1024,
            max_token_size=8192,
            func=partial(
                ollama_embed.func,
                embed_model=os.getenv("EMBEDDING_MODEL"),
                host=os.getenv("EMBEDDING_BINDING_HOST"),
            ),
        ),
    )
    await rag.initialize_storages()

    # Insert documents

    await rag.ainsert("Your document text goes here...")

    # Query with streaming

    resp = await rag.aquery(
        "Summarize the main ideas.",
        param=QueryParam(mode="hybrid", stream=True),
    )
    async for chunk in resp:
        print(chunk, end="", flush=True)

    # Cleanup

    await rag.finalize_storages()

if __name__ == "__main__":
    set_verbose_debug(True)
    asyncio.run(build_rag())

This implementation references the official lightrag_ollama_demo.py example and demonstrates how to wire the ollama_model_complete and ollama_embed functions into the core LightRAG orchestrator.

Summary

• Architecture: LightRAG's Ollama integration (lightrag/llm/ollama.py) provides async wrappers that handle host selection, retries, and resource cleanup automatically.
• Context Control: Pass options={"num_ctx": 8192} in llm_model_kwargs to configure Ollama's context window; LightRAG defers token limit enforcement to the underlying Ollama model.
• Host Flexibility: The _coerce_host_for_cloud_model helper switches to Ollama Cloud automatically when model names end with -cloud or :cloud.
• Embeddings: The ollama_embed function exposes 1024-dimensional vectors with an 8192-token max window, delegating truncation to Ollama's server-side logic.
• Resilience: Built-in Tenacity retry logic (3 attempts, exponential backoff) ensures stable local inference even during resource contention.

Frequently Asked Questions

How do I increase the context window beyond 8K tokens when using LightRAG with Ollama?

Increase the num_ctx value in the options dictionary within llm_model_kwargs. For example, set options={"num_ctx": 32768} for a 32K context window. LightRAG forwards this parameter directly to Ollama's API without enforcing upper limits, though you must ensure your local model (e.g., Llama 3.1, Qwen2.5) actually supports the requested context length.

Can I switch between local Ollama and Ollama Cloud without changing my code?

Yes. The Ollama binding automatically detects cloud models through the _coerce_host_for_cloud_model function. If your llm_model_name ends with -cloud or :cloud, LightRAG switches the host from localhost:11434 to the Ollama Cloud endpoint. Otherwise, it defaults to the local host specified in your configuration.

Why does LightRAG not truncate my texts when using Ollama embeddings?

The ollama_embed function (lines 75-84 in lightrag/llm/ollama.py) forwards the complete text list to ollama.AsyncClient.embed and does not implement client-side truncation. Ollama handles truncation internally based on its own num_ctx settings. If you require strict token limits, configure num_ctx in your Ollama server configuration or pre-process texts before insertion.

How does LightRAG handle connection timeouts with local Ollama servers?

The ollama_model_complete function accepts a timeout parameter (in seconds) through llm_model_kwargs. The implementation uses Tenacity to retry failed requests up to three times with exponential backoff (lines 61-84). Set timeout to 300 or higher when processing large contexts to prevent premature connection drops during inference.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}