Architecture for Integrating Different LLM Providers: Ollama and Gemini Explained

The interviewstreet/hiring-agent repository abstracts LLM back-ends behind a unified interface that exposes a single chat method, enabling the codebase to route requests to either Ollama or Google Gemini without vendor-specific logic.

The architecture decouples model inference from application logic through a three-layer design consisting of a provider enumeration, concrete provider classes, and a central factory. This structure lives primarily in models.py, prompt.py, and llm_utils.py, making it straightforward to swap local Ollama instances with cloud-based Gemini APIs—or add entirely new providers—by updating configuration rather than rewriting business logic.

Three-Layer Provider Abstraction

The integration strategy relies on three distinct layers that separate concerns between type safety, vendor-specific implementations, and runtime selection.

Provider Enumeration

The ModelProvider enum in models.py guarantees a finite set of supported back-ends and enables explicit mapping of model names to their respective handlers.

class ModelProvider(Enum):
    OLLAMA = "ollama"
    GEMINI = "gemini"

This enumeration acts as the single source of truth for provider identity throughout the application. Any code checking provider type compares against these enum values rather than hard-coded strings, preventing drift between configuration and implementation.

Concrete Provider Classes

Each LLM back-end implements a common interface defined by the chat method signature: chat(model, messages, options, **kwargs). Both classes reside in models.py.

OllamaProvider wraps the official ollama Python client and forwards requests with a 32,000-token context window. It accepts the generic message format and transmits it unchanged to the local Ollama server.

GeminiProvider configures google.generativeai with the supplied API key, translates the standard message list into Gemini’s role/parts structure, applies exponential back-off on quota errors, and converts the Gemini response back into an Ollama-style JSON payload. This normalization ensures downstream components receive a consistent data structure regardless of which vendor generated the response.

Factory Selection Logic

The initialize_llm_provider function in llm_utils.py serves as the central dispatcher. It reads the requested model name, consults the provider mapping, validates environment configuration (such as API keys), and returns an instantiated provider class.

def initialize_llm_provider(model_name: str) -> Any:
    provider = OllamaProvider()  # default fallback

    model_provider = MODEL_PROVIDER_MAPPING.get(
        model_name, ModelProvider.OLLAMA
    )
    if model_provider == ModelProvider.GEMINI:
        if not GEMINI_API_KEY:
            logger.warning("⚠️ Gemini API key not found. Falling back to Ollama.")
        else:
            logger.info(f"🔄 Using Google Gemini API provider with model {model_name}")
            provider = GeminiProvider(api_key=GEMINI_API_KEY)
    else:
        logger.info(f"🔄 Using Ollama provider with model {model_name}")
    return provider

This factory guarantees that the rest of the application receives an object with a predictable chat method while centralizing provider-specific instantiation logic.

Model-to-Provider Mapping Configuration

The prompt.py file contains the MODEL_PROVIDER_MAPPING dictionary that explicitly associates each supported model with its provider enum value.

MODEL_PROVIDER_MAPPING = {
    "qwen3:1.7b": ModelProvider.OLLAMA,
    "gemma3:4b":  ModelProvider.OLLAMA,
    "gemini-2.0-flash": ModelProvider.GEMINI,
    # … other models …

}

When initialize_llm_provider receives a model name, it looks up the corresponding ModelProvider value here. If the model is absent from the mapping, the system defaults to ModelProvider.OLLAMA, ensuring robustness against configuration drift.

Environment-Based Configuration and Fallbacks

Configuration flows through environment variables loaded via python-dotenv in prompt.py. The system respects three critical variables:

• DEFAULT_MODEL – Specifies the fallback model when none is explicitly requested.
• LLM_PROVIDER – Overrides the auto-detection mechanism in specific edge cases.
• GEMINI_API_KEY – Required for instantiating GeminiProvider; if missing, the factory logs a warning and returns OllamaProvider instead.

This graceful degradation ensures that development environments without Google Cloud credentials automatically route to local Ollama instances without crashing the application.

Usage Flow and Polymorphic Interface

The consumer code interacts with the architecture through a four-step flow that remains identical regardless of which provider ultimately handles the request:

1. Determine the target model name (e.g., "gemini-2.0-flash" or "gemma3:4b").
2. Call initialize_llm_provider(model_name) to receive a concrete provider instance.
3. Execute provider.chat(model=model_name, messages=msg_list, options=params).
4. Process the standardized response dictionary containing the "message" field.

Because both providers return responses in the same Ollama-compatible JSON structure, downstream components—such as RAG pipelines or evaluation frameworks—remain completely agnostic to the underlying vendor.

Extending the Architecture with New Providers

Adding support for additional LLM services requires three minimal changes:

1. Define the provider enum value in models.py by extending ModelProvider.
2. Implement the provider class in models.py with a chat method matching the signature chat(self, model, messages, options=None, **kwargs).
3. Update the mapping and factory in prompt.py and llm_utils.py to recognize the new provider type and instantiate the class when appropriate.

This plug-in architecture ensures that integrations with providers like Anthropic Claude, OpenAI GPT, or Cohere Command can be added without modifying the core application logic.

Summary

• The three-layer architecture (enumeration, concrete classes, factory) decouples vendor-specific SDKs from business logic in the interviewstreet/hiring-agent repository.
• ModelProvider enum and MODEL_PROVIDER_MAPPING dictionary in prompt.py provide explicit, type-safe routing from model names to implementations.
• initialize_llm_provider in llm_utils.py centralizes instantiation logic, handles environment validation, and implements graceful fallbacks to Ollama when Gemini credentials are absent.
• Both OllamaProvider and GeminiProvider expose identical chat method signatures and normalize responses to a common JSON format.
• The system supports extensible configuration through .env files, allowing developers to switch between local and cloud inference by changing environment variables rather than code.

Frequently Asked Questions

How does the factory decide between Ollama and Gemini?

The initialize_llm_provider function looks up the requested model name in the MODEL_PROVIDER_MAPPING dictionary defined in prompt.py. If the mapped value equals ModelProvider.GEMINI and the GEMINI_API_KEY environment variable is present, it returns a GeminiProvider instance; otherwise, it defaults to OllamaProvider.

What happens if the Gemini API key is missing?

When the factory detects a Gemini-mapped model but finds GEMINI_API_KEY unset or empty, it logs a warning message ("⚠️ Gemini API key not found. Falling back to Ollama.") and returns an OllamaProvider instance instead. This ensures the application continues running using local inference rather than failing with an authentication error.

Can I add custom providers to the architecture?

Yes. Create a new class in models.py that implements the chat(model, messages, options, **kwargs) method, add a corresponding entry to the ModelProvider enum, update MODEL_PROVIDER_MAPPING in prompt.py to map model names to your new enum value, and extend initialize_llm_provider in llm_utils.py to instantiate your class when that enum value is detected.

Where is the model-to-provider mapping defined?

The MODEL_PROVIDER_MAPPING dictionary resides in prompt.py. It maps string model identifiers (such as "gemini-2.0-flash" or "qwen3:1.7b") to their respective ModelProvider enum values, enabling the factory function to resolve which concrete provider class to instantiate at runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}