How the Ollama LLM Provider Works in Hiring Agent: Implementation Guide

The Ollama LLM provider wraps the local Ollama server via the ollama Python package, sanitizing parameters and enforcing a 32,768-token context window to enable fully offline inference without API keys.

The interviewstreet/hiring-agent repository abstracts language model interactions through a provider pattern. The OllamaProvider class serves as the default fallback implementation when no external GEMINI_API_KEY is configured, allowing the application to run against a locally hosted Ollama server using the standardized interface defined in models.py.

Architecture and Provider Selection

The application decouples LLM selection from business logic using a factory function that instantiates the appropriate provider at runtime.

Runtime Provider Selection Logic

The get_llm_provider function in llm_utils.py determines which implementation to use based on environment variables. If GEMINI_API_KEY is present, it returns a GeminiProvider instance; otherwise, it instantiates OllamaProvider. This design allows seamless switching between cloud and local inference without modifying downstream consumers.

# Inside llm_utils.py

from models import OllamaProvider, GeminiProvider
import os

def get_llm_provider():
    # Prefer Gemini if a key is present; otherwise use Ollama

    api_key = os.getenv("GEMINI_API_KEY")
    if api_key:
        return GeminiProvider(api_key)
    else:
        return OllamaProvider()

The OllamaProvider Class

Defined at models.py#L71-L105, this class implements the provider contract. Its __init__ method imports the ollama library and stores it as self.client, while the chat method normalizes requests and handles communication with the local Ollama HTTP server.

How OllamaProvider.chat Works Internally

The chat method executes a four-stage pipeline to prepare and dispatch requests to localhost:11434.

1. Client Initialization – The constructor imports the ollama package and assigns it to self.client, establishing the connection to the local server.

2. Option Sanitization – The method copies the input options dictionary and performs two critical sanitization steps:

   • Removes any stream key, as Ollama's HTTP API does not accept this parameter at the top level
   • Forces num_ctx to 32768 to ensure a large, consistent context window across all models

3. Parameter Assembly – It constructs a chat_params dictionary containing:

   • model: The Ollama model name (e.g., llama2, mistral)
   • messages: The conversation history as a list of {"role": "...", "content": "..."} dictionaries
   • options: The sanitized configuration dictionary
   • Optional keys like stream and format passed via **kwargs

4. Request Dispatch – The method calls self.client.chat(**chat_params), which performs an HTTP POST to the Ollama server. The raw JSON response is returned directly, maintaining compatibility with the response format expected by other providers.

Implementation Examples

Direct Provider Usage

When working outside the standard evaluator flow, you can instantiate OllamaProvider directly to send chat requests to your local server:

from models import OllamaProvider

# Initialise provider (no API key required)

ollama = OllamaProvider()

# Example conversation

messages = [
    {"role": "system", "content": "You are a helpful interview coach."},
    {"role": "user", "content": "Give me feedback on my resume summary."},
]

# Optional generation options

options = {"temperature": 0.7, "top_p": 0.9}

# Perform the chat request

response = ollama.chat(
    model="llama2",          # any model available to your local Ollama server

    messages=messages,
    options=options,
    stream=False,           # optional: disable streaming

)

print(response["message"]["content"])

Integration in the Evaluation Pipeline

Higher-level modules like evaluator.py consume the provider through the factory function, remaining agnostic to the underlying implementation:

# evaluator.py (simplified)

def evaluate_resume(resume_text):
    provider = get_llm_provider()
    prompt = build_prompt(resume_text)        # uses Jinja templates

    response = provider.chat(
        model="llama2",
        messages=[{"role": "user", "content": prompt}],
        options={"temperature": 0.0},
    )
    return response["message"]["content"]

Key Source Files

• models.py – Contains the OllamaProvider class implementation and the GeminiProvider counterpart (source)
• llm_utils.py – Houses the get_llm_provider factory function for runtime selection logic (source)
• evaluator.py – Example consumer that obtains LLM feedback for resume evaluation (source)
• prompt.py – Assembles Jinja-templated messages passed to the provider's chat method (source)

Summary

• The OllamaProvider class in models.py enables local LLM inference by wrapping the official ollama Python package.
• Option sanitization automatically removes unsupported stream parameters and enforces a 32,768-token context window via the num_ctx setting.
• The get_llm_provider function in llm_utils.py selects OllamaProvider automatically when no GEMINI_API_KEY environment variable is present.
• Downstream modules remain provider-agnostic, receiving standardized response dictionaries that match the shape of Gemini API responses.
• All communication occurs via HTTP POST to the local Ollama server (typically localhost:11434/api/chat).

Frequently Asked Questions

What is the default LLM provider in Hiring Agent?

When the GEMINI_API_KEY environment variable is unset, the application defaults to OllamaProvider according to the logic in llm_utils.py. This fallback mechanism ensures the hiring agent can operate entirely offline using locally hosted models without requiring cloud API credentials.

How does the Ollama provider handle streaming responses?

The provider explicitly removes any stream key from the options dictionary during the sanitization phase before dispatching to Ollama's HTTP API. While the underlying ollama Python package supports streaming, the Hiring Agent implementation strips this parameter to maintain compatibility with the synchronous response format expected by the evaluator and other consumers.

What context window size does the Ollama provider use?

According to the source code in models.py, the provider forces a num_ctx value of 32768 tokens during the option sanitization step. This hardcoded default ensures consistent, large context windows across all models regardless of their individual default configurations, supporting evaluation of lengthy resume content.

Can I use OllamaProvider with any local model?

Yes, the model parameter passed to chat() accepts any model name available to your local Ollama server, including llama2, mistral, codellama, or custom fine-tuned models. The provider passes this string directly to the Ollama HTTP API endpoint without validation, allowing immediate access to any model currently pulled in your local Ollama registry.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}