How Model Selection Works When Multiple AI Providers Are Supported in screenshot-to-code

The screenshot-to-code repository implements a deterministic, rule-based model selection stage that evaluates available API keys (OpenAI, Anthropic, and Gemini) alongside request parameters to dynamically choose the optimal LLM combinations from predefined constant tuples.

The abi/screenshot-to-code project orchestrates code generation across diverse AI providers through an intelligent routing mechanism. When processing requests, the backend must decide whether to invoke OpenAI's GPT models, Anthropic's Claude, or Google's Gemini—often mixing providers to maximize output quality. This article examines the ModelSelectionStage class in backend/routes/generate_code.py to reveal exactly how the system prioritizes providers and balances workload across multiple LLMs.

The ModelSelectionStage Entry Point

When a generation request arrives, the backend instantiates a ModelSelectionStage object and calls its asynchronous select_models method. This method serves as the primary entry point for all model selection decisions, receiving the generation type ("create" or "update"), the input mode ("text", "image", or "video"), and the three possible API keys (OPENAI_API_KEY, ANTHROPIC_API_KEY, GEMINI_API_KEY).

The method determines the number of variants to generate using NUM_VARIANTS (default 4) for standard requests or NUM_VARIANTS_VIDEO (default 2) for video mode, then delegates the actual provider selection to a private helper named _get_variant_models. According to the source code in backend/routes/generate_code.py (lines 53-71), this architecture cleanly separates request validation from provider-specific logic.

Provider Prioritization Logic in _get_variant_models

The _get_variant_models helper contains the complete decision matrix for selecting models based on available API keys. Located in backend/routes/generate_code.py, this method evaluates conditions in priority order to determine which predefined model tuple to return from backend/routes/model_choice_sets.py.

Video Mode Requirements

If input_mode == "video", the system strictly requires a Gemini API key and returns VIDEO_VARIANT_MODELS. This mode exclusively uses Google's Gemini models to process video inputs, raising an error if the Gemini key is missing.

Edit/Update Mode Optimization

When the generation type is an edit or update and both Gemini and OpenAI keys are present, the system prefers one Gemini and one fast OpenAI model. Specifically, it selects GEMINI_3_FLASH_PREVIEW_MINIMAL paired with GPT_5_2_CODEX_LOW to balance quality with responsiveness for iterative editing tasks.

Multi-Provider Scenarios

If all three API keys are present, the method selects from ALL_KEYS_MODELS_* constants depending on context. For text creation tasks, it uses ALL_KEYS_MODELS_TEXT_CREATE (typically including Gemini Flash minimal, GPT‑5.2 Codex high, Claude Opus, and Gemini 3 Pro preview low). For updates, it uses ALL_KEYS_MODELS_UPDATE, falling back to ALL_KEYS_MODELS_DEFAULT for other cases.

Fallback Strategies

When only two keys are available, the system uses paired constants: GEMINI_ANTHROPIC_MODELS, GEMINI_OPENAI_MODELS, or OPENAI_ANTHROPIC_MODELS. If only a single provider is configured, it falls back to provider-specific tuples like OPENAI_ONLY_MODELS (containing high and medium Codex variants), ANTHROPIC_ONLY_MODELS, or GEMINI_ONLY_MODELS. When no keys are detected, the stage raises an exception prompting the user to add an API key.

Round-Robin Variant Distribution

After selecting the appropriate model tuple, the system must ensure the number of variants matches the requested count. The implementation cycles through the chosen model list using modulo arithmetic:

selected_models = []
for i in range(num_variants):
    selected_models.append(models[i % len(models)])

This round-robin approach, found in backend/routes/generate_code.py (lines 44-48), guarantees that if the requested variant count exceeds the available models, the system distributes work evenly across providers. For example, with two models but four variants requested, each model handles two generation tasks.

Configuration Constants and Defaults

The variant counts are controlled by global constants defined in backend/config.py. NUM_VARIANTS defaults to 4 for standard image and text generations, while NUM_VARIANTS_VIDEO defaults to 2 for video processing tasks. These values drive how many times the model selection logic cycles through the available provider tuples, directly impacting the number of alternative code generations the user receives.

Error Handling for Missing API Keys

The selection stage validates API key presence before attempting model assignment. If required keys are missing—for instance, attempting video generation without a Gemini key—the _get_variant_models method raises an exception that bubbles up to the HTTP handler in backend/routes/generate_code.py (lines 84-92). The handler surfaces clear error messages to the client, preventing cryptic failures later in the generation pipeline.

Practical Implementation Examples

Using the Selector with All Providers

This example demonstrates model selection when all three API keys are configured for a text creation task:

from backend.routes.generate_code import ModelSelectionStage
from backend.routes.model_choice_sets import (
    VIDEO_VARIANT_MODELS, ALL_KEYS_MODELS_DEFAULT,
)
import asyncio

async def demo():
    # Simulate an environment where all three keys are present

    selector = ModelSelectionStage(throw_error=lambda msg: asyncio.sleep(0))
    models = await selector.select_models(
        generation_type="create",
        input_mode="text",            # could be "image" or "video"

        openai_api_key="sk-...",      # any non‑None value

        anthropic_api_key="sk-ant...",# any non‑None value

        gemini_api_key="sk-gem...",   # any non‑None value

    )
    print([m.value for m in models])

asyncio.run(demo())

When executed with generation_type="create" and input_mode="text", this returns the ALL_KEYS_MODELS_TEXT_CREATE tuple containing four distinct models across the three providers.

Video Mode with Gemini Only

Video processing requires Gemini access and ignores other providers:

async def video_demo():
    selector = ModelSelectionStage(throw_error=lambda msg: asyncio.sleep(0))
    models = await selector.select_models(
        generation_type="create",
        input_mode="video",
        openai_api_key=None,
        anthropic_api_key=None,
        gemini_api_key="sk-gemini",   # only Gemini key needed

    )
    # Will be VIDEO_VARIANT_MODELS (two Gemini models)

    print([m.value for m in models])

asyncio.run(video_demo())

OpenAI-Only Fallback

When only OpenAI is configured, the system automatically restricts itself to OpenAI models:

async def openai_only_demo():
    selector = ModelSelectionStage(throw_error=lambda msg: asyncio.sleep(0))
    models = await selector.select_models(
        generation_type="update",
        input_mode="image",
        openai_api_key="sk-openai",
        anthropic_api_key=None,
        gemini_api_key=None,
    )
    # Returns the OPENAI_ONLY_MODELS tuple (high & medium Codex)

    print([m.value for m in models])

asyncio.run(openai_only_demo())

Summary

• Model selection occurs through ModelSelectionStage.select_models in backend/routes/generate_code.py, which evaluates request context and available API keys.
• Decision logic resides in _get_variant_models, using a priority matrix to select from constant tuples defined in backend/routes/model_choice_sets.py.
• Video mode strictly requires Gemini API keys and uses VIDEO_VARIANT_MODELS.
• Variant distribution employs round-robin cycling to match NUM_VARIANTS (4) or NUM_VARIANTS_VIDEO (2) from backend/config.py.
• Error handling validates key presence before processing, surfacing clear messages for missing required providers.

Frequently Asked Questions

What happens if I only provide one AI provider's API key?

The system falls back to provider-specific model tuples defined in backend/routes/model_choice_sets.py. For example, with only an OpenAI key, it selects OPENAI_ONLY_MODELS containing high and medium Codex variants, ensuring the application functions with single-provider configurations.

Why does video input mode require a Gemini API key?

Video processing relies on Gemini's multimodal capabilities for analyzing video content. The _get_variant_models function explicitly checks for input_mode == "video" and returns VIDEO_VARIANT_MODELS only when a Gemini key is present, as other providers currently lack the necessary video understanding capabilities for this workflow.

How does the system determine how many model variants to generate?

The variant count comes from NUM_VARIANTS (default 4) for standard generations or NUM_VARIANTS_VIDEO (default 2) for video mode, both defined in backend/config.py. These constants control how many times the selected models cycle to produce alternative code generations for a single request.

Can I customize which specific models are used for each provider combination?

Yes, by modifying the constant tuples in backend/routes/model_choice_sets.py (lines 1-68). Each tuple—such as ALL_KEYS_MODELS_TEXT_CREATE or GEMINI_ANTHROPIC_MODELS—contains specific Llm enum values from backend/llm.py that you can adjust to use different model variants like GPT-4, Claude Opus, or Gemini Pro.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}