# How to Customize Ollama Model and Temperature Settings in the Agentic RAG Pipeline

> Customize Ollama model and temperature settings in your RAG pipeline. Learn how to adjust global, service-wide, and per-request configurations for optimal results.

- Repository: [jamwithai/production-agentic-rag-course](https://github.com/jamwithai/production-agentic-rag-course)
- Tags: how-to-guide
- Published: 2026-03-23

---

**You can customize the Ollama model and temperature at three levels—global defaults via environment variables, service-wide through `GraphConfig`, or per-request using the `ask()` method—in the production-agentic-rag-course repository.**

The **Agentic RAG** service in the `jamwithai/production-agentic-rag-course` repository uses Ollama as its local LLM backend. Both the model that Ollama runs and the generation temperature are configurable through a hierarchical system that flows from configuration files to runtime context and finally to the Ollama API call.

## Understanding the Configuration Hierarchy

The system provides three distinct scopes for customizing LLM behavior, each suited for different deployment scenarios.

### Global Defaults (Environment Variables)

Global defaults originate in [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) and [`src/services/agents/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/config.py). The `Settings` class reads from your environment (typically a `.env` file) to establish baseline values.

In [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) (lines 73-75), the `Settings` class defines:

```python
ollama_host: str = "http://localhost:11434"
ollama_model: str = "llama3.2:1b"
ollama_timeout: int = 30

```

Meanwhile, [`src/services/agents/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/config.py) (lines 27-29) provides the `GraphConfig` dataclass used throughout the workflow:

```python
model: str = "llama3.2:1b"
temperature: float = 0.0

```

These values populate the runtime `Context` when the service initializes.

### Service-Wide Overrides (GraphConfig)

When constructing the `AgenticRAGService` or using the factory function `make_agentic_rag_service`, you can inject a custom `GraphConfig` that replaces the defaults. This configuration is stored in `self.graph_config` and applies to all subsequent operations unless overridden per-request.

### Per-Request Overrides (ask() Method)

The `ask()` method in `AgenticRAGService` accepts a `model` argument that temporarily overrides `self.graph_config.model` for that specific call. Note that temperature cannot be overridden at this level; it remains as defined in the active `GraphConfig`.

## How Settings Flow Through the Pipeline

Understanding the data flow helps debug configuration issues. The path from configuration to Ollama API follows these stages:

1. **Context Creation**: In [`src/services/agents/agentic_rag.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/agentic_rag.py) (lines 52-61), the `_run_workflow` method instantiates a `Context` object, pulling `model_name` and `temperature` directly from `self.graph_config`.

2. **Node Execution**: Generation nodes like [`generate_answer_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/generate_answer_node.py) (lines 83-86) access these values through the runtime context:

```python
llm = runtime.context.ollama_client.get_langchain_model(
    model=runtime.context.model_name,
    temperature=runtime.context.temperature,
)

```

3. **API Transmission**: Finally, [`src/services/ollama/client.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/ollama/client.py) (lines 21-23) sends these parameters to Ollama's `/api/generate` endpoint as JSON payload fields `model` and `temperature`.

## Customizing the Ollama Model

You can specify which Ollama model runs at each configuration level.

### Method 1: Environment Variable Configuration

Add to your `.env` file or export before running:

```dotenv
OLLAMA_HOST=http://localhost:11434
OLLAMA_MODEL=phi3:latest
OLLAMA_TIMEOUT=30

```

These values are read by the `Settings` class in [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) and become the default unless overridden later in the initialization chain.

### Method 2: Service Initialization with GraphConfig

Pass a custom `GraphConfig` when building the service:

```python
from src.services.agents.config import GraphConfig
from src.services.agents.agentic_rag import AgenticRAGService

# Assuming required clients are initialized

custom_cfg = GraphConfig(model="phi3:latest", temperature=0.6)

service = AgenticRAGService(
    opensearch_client=opensearch_client,
    ollama_client=ollama_client,
    embeddings_client=embeddings_client,
    graph_config=custom_cfg,
)

```

Alternatively, using the factory pattern:

```python
from src.services.agents.factory import make_agentic_rag_service

service = make_agentic_rag_service(
    opensearch_client,
    ollama_client,
    embeddings_client,
)
service.graph_config.model = "phi3:latest"
service.graph_config.temperature = 0.6

```

### Method 3: Per-Request Model Override

Override the model for a single query without affecting service defaults:

```python
answer = await service.ask(
    "What is Retrieval-Augmented Generation?",
    model="mistral:instruct",  # Temporary override

)

```

**Important**: The `ask()` method only accepts `model` as a per-request override. Temperature cannot be changed here without modifying the service's `graph_config` temporarily.

## Adjusting the Temperature Setting

Temperature controls generation randomness (0.0 = deterministic, higher = more creative). Unlike the model name, temperature can only be set at the global or service-wide levels.

### Service-Wide Temperature Configuration

Set temperature during `GraphConfig` initialization:

```python
graph_cfg = GraphConfig(
    model="llama3.2:1b",
    temperature=0.7,  # More creative output

)

```

### Temporary Temperature Change for Single Requests

Since `ask()` does not accept a temperature parameter, modify `service.graph_config.temperature` before calling:

```python

# Preserve original temperature

prev_temp = service.graph_config.temperature

# Set temporary temperature

service.graph_config.temperature = 0.8
answer = await service.ask("Tell me a creative story about robots.")

# Restore original value if needed

service.graph_config.temperature = prev_temp

```

All nodes that invoke Ollama read `runtime.context.temperature`, ensuring your chosen value reaches the Ollama `/api/generate` payload.

## Complete Working Example

This example demonstrates setting environment variables, configuring a custom `GraphConfig`, and running a query:

```python
import os
from src.services.agents.config import GraphConfig
from src.services.agents.factory import make_agentic_rag_service
from src.services.opensearch.client import OpenSearchClient
from src.services.ollama.client import OllamaClient
from src.services.embeddings.jina_client import JinaEmbeddingsClient

# 1. Configure environment (optional)

os.environ["OLLAMA_HOST"] = "http://localhost:11434"
os.environ["OLLAMA_MODEL"] = "phi3:latest"

# 2. Initialize clients (implementation details omitted)

opensearch = OpenSearchClient(...)
ollama = OllamaClient(...)
embeddings = JinaEmbeddingsClient(...)

# 3. Create custom configuration

graph_cfg = GraphConfig(
    model="phi3:latest",
    temperature=0.7,
    top_k=5,
)

# 4. Build service with custom config

service = make_agentic_rag_service(
    opensearch_client=opensearch,
    ollama_client=ollama,
    embeddings_client=embeddings,
    top_k=graph_cfg.top_k,
    use_hybrid=True,
)
service.graph_config = graph_cfg

# 5. Execute query

result = await service.ask("Explain dense versus sparse retrieval.")
print(result["answer"])
print("Sources:", result["sources"])

```

In this flow, the `Context` created in `_run_workflow` carries `model="phi3:latest"` and `temperature=0.7`, which [`generate_answer_node.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/generate_answer_node.py) passes to the Ollama client, ultimately appearing in the HTTP POST to `/api/generate`.

## Summary

- **Global defaults** are set via `.env` files through `Settings` in [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) and `GraphConfig` defaults in [`src/services/agents/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/config.py).
- **Service-wide customization** occurs by passing a custom `GraphConfig` to `AgenticRAGService` or modifying `service.graph_config` directly.
- **Per-request model overrides** are available via the `model` parameter in `ask()`, but temperature requires temporarily modifying `service.graph_config.temperature` before the call.
- The runtime `Context` in [`src/services/agents/context.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/context.py) propagates these values to generation nodes, which pass them to the Ollama client for API calls.

## Frequently Asked Questions

### Can I override the temperature for a single request without affecting other users?

Not directly through the `ask()` method signature. You must temporarily modify `service.graph_config.temperature` before calling `ask()` and restore it afterward. The `ask()` method only accepts a `model` parameter for per-request overrides, while temperature remains bound to the `GraphConfig` instance.

### Where does the system validate the model name?

The repository does not appear to perform explicit validation of model names against Ollama's available models. The `model` string (defaulting to `"llama3.2:1b"` in `GraphConfig`) is passed directly to the Ollama client's `get_langchain_model` method and ultimately to the `/api/generate` endpoint. Ensure your specified model is already pulled and available in your Ollama instance.

### What happens if I don't specify any model configuration?

The system falls back to the defaults defined in [`src/services/agents/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/services/agents/config.py) (`model="llama3.2:1b"`, `temperature=0.0`). If you've set `OLLAMA_MODEL` in your environment, that value populates through `Settings` in [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) and typically becomes the default when the service initializes via standard factory patterns.

### How do I switch between different Ollama hosts dynamically?

The Ollama host is configured in [`src/config.py`](https://github.com/jamwithai/production-agentic-rag-course/blob/main/src/config.py) via the `OLLAMA_HOST` environment variable (defaulting to `http://localhost:11434`). To switch hosts dynamically, you would need to reinitialize the `OllamaClient` with a new base URL or restart the service with updated environment variables, as the client instance is typically injected into the service at construction time.