How to Customize Ollama Model and Temperature Settings in the Agentic RAG Pipeline

You can customize the Ollama model and temperature at three levels—global defaults via environment variables, service-wide through GraphConfig, or per-request using the ask() method—in the production-agentic-rag-course repository.

The Agentic RAG service in the jamwithai/production-agentic-rag-course repository uses Ollama as its local LLM backend. Both the model that Ollama runs and the generation temperature are configurable through a hierarchical system that flows from configuration files to runtime context and finally to the Ollama API call.

Understanding the Configuration Hierarchy

The system provides three distinct scopes for customizing LLM behavior, each suited for different deployment scenarios.

Global Defaults (Environment Variables)

Global defaults originate in src/config.py and src/services/agents/config.py. The Settings class reads from your environment (typically a .env file) to establish baseline values.

In src/config.py (lines 73-75), the Settings class defines:

ollama_host: str = "http://localhost:11434"
ollama_model: str = "llama3.2:1b"
ollama_timeout: int = 30

Meanwhile, src/services/agents/config.py (lines 27-29) provides the GraphConfig dataclass used throughout the workflow:

model: str = "llama3.2:1b"
temperature: float = 0.0

These values populate the runtime Context when the service initializes.

Service-Wide Overrides (GraphConfig)

When constructing the AgenticRAGService or using the factory function make_agentic_rag_service, you can inject a custom GraphConfig that replaces the defaults. This configuration is stored in self.graph_config and applies to all subsequent operations unless overridden per-request.

Per-Request Overrides (ask() Method)

The ask() method in AgenticRAGService accepts a model argument that temporarily overrides self.graph_config.model for that specific call. Note that temperature cannot be overridden at this level; it remains as defined in the active GraphConfig.

How Settings Flow Through the Pipeline

Understanding the data flow helps debug configuration issues. The path from configuration to Ollama API follows these stages:

1. Context Creation: In src/services/agents/agentic_rag.py (lines 52-61), the _run_workflow method instantiates a Context object, pulling model_name and temperature directly from self.graph_config.

2. Node Execution: Generation nodes like generate_answer_node.py (lines 83-86) access these values through the runtime context:

llm = runtime.context.ollama_client.get_langchain_model(
    model=runtime.context.model_name,
    temperature=runtime.context.temperature,
)

3. API Transmission: Finally, src/services/ollama/client.py (lines 21-23) sends these parameters to Ollama's /api/generate endpoint as JSON payload fields model and temperature.

Customizing the Ollama Model

You can specify which Ollama model runs at each configuration level.

Method 1: Environment Variable Configuration

Add to your .env file or export before running:

OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=phi3:latest
OLLAMA_TIMEOUT=30

These values are read by the Settings class in src/config.py and become the default unless overridden later in the initialization chain.

Method 2: Service Initialization with GraphConfig

Pass a custom GraphConfig when building the service:

from src.services.agents.config import GraphConfig
from src.services.agents.agentic_rag import AgenticRAGService

# Assuming required clients are initialized

custom_cfg = GraphConfig(model="phi3:latest", temperature=0.6)

service = AgenticRAGService(
    opensearch_client=opensearch_client,
    ollama_client=ollama_client,
    embeddings_client=embeddings_client,
    graph_config=custom_cfg,
)

Alternatively, using the factory pattern:

from src.services.agents.factory import make_agentic_rag_service

service = make_agentic_rag_service(
    opensearch_client,
    ollama_client,
    embeddings_client,
)
service.graph_config.model = "phi3:latest"
service.graph_config.temperature = 0.6

Method 3: Per-Request Model Override

Override the model for a single query without affecting service defaults:

answer = await service.ask(
    "What is Retrieval-Augmented Generation?",
    model="mistral:instruct",  # Temporary override

)

Important: The ask() method only accepts model as a per-request override. Temperature cannot be changed here without modifying the service's graph_config temporarily.

Adjusting the Temperature Setting

Temperature controls generation randomness (0.0 = deterministic, higher = more creative). Unlike the model name, temperature can only be set at the global or service-wide levels.

Service-Wide Temperature Configuration

Set temperature during GraphConfig initialization:

graph_cfg = GraphConfig(
    model="llama3.2:1b",
    temperature=0.7,  # More creative output

)

Temporary Temperature Change for Single Requests

Since ask() does not accept a temperature parameter, modify service.graph_config.temperature before calling:

# Preserve original temperature

prev_temp = service.graph_config.temperature

# Set temporary temperature

service.graph_config.temperature = 0.8
answer = await service.ask("Tell me a creative story about robots.")

# Restore original value if needed

service.graph_config.temperature = prev_temp

All nodes that invoke Ollama read runtime.context.temperature, ensuring your chosen value reaches the Ollama /api/generate payload.

Complete Working Example

This example demonstrates setting environment variables, configuring a custom GraphConfig, and running a query:

import os
from src.services.agents.config import GraphConfig
from src.services.agents.factory import make_agentic_rag_service
from src.services.opensearch.client import OpenSearchClient
from src.services.ollama.client import OllamaClient
from src.services.embeddings.jina_client import JinaEmbeddingsClient

# 1. Configure environment (optional)

os.environ["OLLAMA_HOST"] = "http://localhost:11434"
os.environ["OLLAMA_MODEL"] = "phi3:latest"

# 2. Initialize clients (implementation details omitted)

opensearch = OpenSearchClient(...)
ollama = OllamaClient(...)
embeddings = JinaEmbeddingsClient(...)

# 3. Create custom configuration

graph_cfg = GraphConfig(
    model="phi3:latest",
    temperature=0.7,
    top_k=5,
)

# 4. Build service with custom config

service = make_agentic_rag_service(
    opensearch_client=opensearch,
    ollama_client=ollama,
    embeddings_client=embeddings,
    top_k=graph_cfg.top_k,
    use_hybrid=True,
)
service.graph_config = graph_cfg

# 5. Execute query

result = await service.ask("Explain dense versus sparse retrieval.")
print(result["answer"])
print("Sources:", result["sources"])

In this flow, the Context created in _run_workflow carries model="phi3:latest" and temperature=0.7, which generate_answer_node.py passes to the Ollama client, ultimately appearing in the HTTP POST to /api/generate.

Summary

• Global defaults are set via .env files through Settings in src/config.py and GraphConfig defaults in src/services/agents/config.py.
• Service-wide customization occurs by passing a custom GraphConfig to AgenticRAGService or modifying service.graph_config directly.
• Per-request model overrides are available via the model parameter in ask(), but temperature requires temporarily modifying service.graph_config.temperature before the call.
• The runtime Context in src/services/agents/context.py propagates these values to generation nodes, which pass them to the Ollama client for API calls.

Frequently Asked Questions

Can I override the temperature for a single request without affecting other users?

Not directly through the ask() method signature. You must temporarily modify service.graph_config.temperature before calling ask() and restore it afterward. The ask() method only accepts a model parameter for per-request overrides, while temperature remains bound to the GraphConfig instance.

Where does the system validate the model name?

The repository does not appear to perform explicit validation of model names against Ollama's available models. The model string (defaulting to "llama3.2:1b" in GraphConfig) is passed directly to the Ollama client's get_langchain_model method and ultimately to the /api/generate endpoint. Ensure your specified model is already pulled and available in your Ollama instance.

What happens if I don't specify any model configuration?

The system falls back to the defaults defined in src/services/agents/config.py (model="llama3.2:1b", temperature=0.0). If you've set OLLAMA_MODEL in your environment, that value populates through Settings in src/config.py and typically becomes the default when the service initializes via standard factory patterns.

How do I switch between different Ollama hosts dynamically?

The Ollama host is configured in src/config.py via the OLLAMA_HOST environment variable (defaulting to http://localhost:11434). To switch hosts dynamically, you would need to reinitialize the OllamaClient with a new base URL or restart the service with updated environment variables, as the client instance is typically injected into the service at construction time.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}