# How the Ollama LLM Provider Works in Hiring Agent: Implementation Guide

> Discover how the Ollama LLM provider works within Hiring Agent. Learn about its implementation for fully offline inference using the ollama Python package without requiring API keys.

- Repository: [HackerRank/hiring-agent](https://github.com/interviewstreet/hiring-agent)
- Tags: implementation-guide
- Published: 2026-07-11

---

**The Ollama LLM provider wraps the local Ollama server via the `ollama` Python package, sanitizing parameters and enforcing a 32,768-token context window to enable fully offline inference without API keys.**

The interviewstreet/hiring-agent repository abstracts language model interactions through a provider pattern. The `OllamaProvider` class serves as the default fallback implementation when no external `GEMINI_API_KEY` is configured, allowing the application to run against a locally hosted Ollama server using the standardized interface defined in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py).

## Architecture and Provider Selection

The application decouples LLM selection from business logic using a factory function that instantiates the appropriate provider at runtime.

### Runtime Provider Selection Logic

The `get_llm_provider` function in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) determines which implementation to use based on environment variables. If `GEMINI_API_KEY` is present, it returns a `GeminiProvider` instance; otherwise, it instantiates `OllamaProvider`. This design allows seamless switching between cloud and local inference without modifying downstream consumers.

```python

# Inside llm_utils.py

from models import OllamaProvider, GeminiProvider
import os

def get_llm_provider():
    # Prefer Gemini if a key is present; otherwise use Ollama

    api_key = os.getenv("GEMINI_API_KEY")
    if api_key:
        return GeminiProvider(api_key)
    else:
        return OllamaProvider()

```

### The OllamaProvider Class

Defined at [`models.py#L71-L105`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py#L71-L105), this class implements the provider contract. Its `__init__` method imports the `ollama` library and stores it as `self.client`, while the `chat` method normalizes requests and handles communication with the local Ollama HTTP server.

## How OllamaProvider.chat Works Internally

The `chat` method executes a four-stage pipeline to prepare and dispatch requests to `localhost:11434`.

1. **Client Initialization** – The constructor imports the `ollama` package and assigns it to `self.client`, establishing the connection to the local server.

2. **Option Sanitization** – The method copies the input `options` dictionary and performs two critical sanitization steps:
   - Removes any `stream` key, as Ollama's HTTP API does not accept this parameter at the top level
   - Forces `num_ctx` to **32768** to ensure a large, consistent context window across all models

3. **Parameter Assembly** – It constructs a `chat_params` dictionary containing:
   - `model`: The Ollama model name (e.g., `llama2`, `mistral`)
   - `messages`: The conversation history as a list of `{"role": "...", "content": "..."}` dictionaries
   - `options`: The sanitized configuration dictionary
   - Optional keys like `stream` and `format` passed via `**kwargs`

4. **Request Dispatch** – The method calls `self.client.chat(**chat_params)`, which performs an HTTP POST to the Ollama server. The raw JSON response is returned directly, maintaining compatibility with the response format expected by other providers.

## Implementation Examples

### Direct Provider Usage

When working outside the standard evaluator flow, you can instantiate `OllamaProvider` directly to send chat requests to your local server:

```python
from models import OllamaProvider

# Initialise provider (no API key required)

ollama = OllamaProvider()

# Example conversation

messages = [
    {"role": "system", "content": "You are a helpful interview coach."},
    {"role": "user", "content": "Give me feedback on my resume summary."},
]

# Optional generation options

options = {"temperature": 0.7, "top_p": 0.9}

# Perform the chat request

response = ollama.chat(
    model="llama2",          # any model available to your local Ollama server

    messages=messages,
    options=options,
    stream=False,           # optional: disable streaming

)

print(response["message"]["content"])

```

### Integration in the Evaluation Pipeline

Higher-level modules like [`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py) consume the provider through the factory function, remaining agnostic to the underlying implementation:

```python

# evaluator.py (simplified)

def evaluate_resume(resume_text):
    provider = get_llm_provider()
    prompt = build_prompt(resume_text)        # uses Jinja templates

    response = provider.chat(
        model="llama2",
        messages=[{"role": "user", "content": prompt}],
        options={"temperature": 0.0},
    )
    return response["message"]["content"]

```

## Key Source Files

- **[`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py)** – Contains the `OllamaProvider` class implementation and the `GeminiProvider` counterpart ([source](https://github.com/interviewstreet/hiring-agent/blob/main/models.py))
- **[`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py)** – Houses the `get_llm_provider` factory function for runtime selection logic ([source](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py))
- **[`evaluator.py`](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py)** – Example consumer that obtains LLM feedback for resume evaluation ([source](https://github.com/interviewstreet/hiring-agent/blob/main/evaluator.py))
- **[`prompt.py`](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py)** – Assembles Jinja-templated messages passed to the provider's `chat` method ([source](https://github.com/interviewstreet/hiring-agent/blob/main/prompt.py))

## Summary

- The `OllamaProvider` class in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py) enables local LLM inference by wrapping the official `ollama` Python package.
- **Option sanitization** automatically removes unsupported `stream` parameters and enforces a **32,768-token context window** via the `num_ctx` setting.
- The `get_llm_provider` function in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py) selects `OllamaProvider` automatically when no `GEMINI_API_KEY` environment variable is present.
- Downstream modules remain provider-agnostic, receiving standardized response dictionaries that match the shape of Gemini API responses.
- All communication occurs via HTTP POST to the local Ollama server (typically `localhost:11434/api/chat`).

## Frequently Asked Questions

### What is the default LLM provider in Hiring Agent?

When the `GEMINI_API_KEY` environment variable is unset, the application defaults to `OllamaProvider` according to the logic in [`llm_utils.py`](https://github.com/interviewstreet/hiring-agent/blob/main/llm_utils.py). This fallback mechanism ensures the hiring agent can operate entirely offline using locally hosted models without requiring cloud API credentials.

### How does the Ollama provider handle streaming responses?

The provider explicitly removes any `stream` key from the options dictionary during the sanitization phase before dispatching to Ollama's HTTP API. While the underlying `ollama` Python package supports streaming, the Hiring Agent implementation strips this parameter to maintain compatibility with the synchronous response format expected by the evaluator and other consumers.

### What context window size does the Ollama provider use?

According to the source code in [`models.py`](https://github.com/interviewstreet/hiring-agent/blob/main/models.py), the provider forces a `num_ctx` value of **32768** tokens during the option sanitization step. This hardcoded default ensures consistent, large context windows across all models regardless of their individual default configurations, supporting evaluation of lengthy resume content.

### Can I use OllamaProvider with any local model?

Yes, the `model` parameter passed to `chat()` accepts any model name available to your local Ollama server, including `llama2`, `mistral`, `codellama`, or custom fine-tuned models. The provider passes this string directly to the Ollama HTTP API endpoint without validation, allowing immediate access to any model currently pulled in your local Ollama registry.