# How the Provider Factory Creates Sessions for OpenAI, Anthropic, and Gemini in screenshot-to-code

> Learn how the screenshot-to-code provider factory creates AI sessions for OpenAI, Anthropic, and Gemini using enum checks for unique authentication and tool serialization.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The `create_provider_session` factory function in `abi/screenshot-to-code` uses enum-based membership checks against provider-specific model sets to instantiate the correct client session, handling authentication and tool serialization uniquely for each AI provider.**

The `abi/screenshot-to-code` repository abstracts LLM provider complexity through a centralized factory pattern. Located in [`backend/agent/providers/factory.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/factory.py), the `create_provider_session` function dynamically routes requests to the appropriate provider implementation based on the `Llm` enum value, eliminating the need for caller-side provider logic while standardizing tool handling and streaming interfaces.

## Model Enumeration and Provider Classification

The foundation of the factory pattern resides in **[`backend/llm.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/llm.py)** (lines 5-70), where the `Llm` enum defines all supported models such as `gpt-4.1-2025-04-14`, `claude-sonnet-4-6`, and `gemini-3-flash-preview`. 

The module maintains a `MODEL_PROVIDER` dictionary that maps every enum member to its provider string (`"openai"`, `"anthropic"`, or `"gemini"`). From this mapping, the system generates convenience sets—`OPENAI_MODELS`, `ANTHROPIC_MODELS`, and `GEMINI_MODELS`—that the factory uses for fast membership checking. This design allows the factory to determine provider affiliation in constant time by checking `if model in OPENAI_MODELS` rather than parsing string values.

## Factory Logic and Session Instantiation

The **`create_provider_session`** function in [`backend/agent/providers/factory.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/factory.py) (lines 16-65) serves as the central instantiation point. It accepts a model enum, prompt messages, image generation flags, and provider-specific API keys, then returns a concrete `ProviderSession` subclass.

```python
def create_provider_session(
    model: Llm,
    prompt_messages: list[ChatCompletionMessageParam],
    should_generate_images: bool,
    openai_api_key: Optional[str],
    openai_base_url: Optional[str],
    anthropic_api_key: Optional[str],
    gemini_api_key: Optional[str],
) -> ProviderSession:
    canonical_tools = canonical_tool_definitions(
        image_generation_enabled=should_generate_images
    )
    # OpenAI

    if model in OPENAI_MODELS:
        if not openai_api_key: raise Exception("OpenAI API key is missing.")
        client = AsyncOpenAI(api_key=openai_api_key, base_url=openai_base_url)
        return OpenAIProviderSession(
            client=client,
            model=model,
            prompt_messages=prompt_messages,
            tools=serialize_openai_tools(canonical_tools),
        )
    # Anthropic

    if model in ANTHROPIC_MODELS:
        if not anthropic_api_key: raise Exception("Anthropic API key is missing.")
        client = AsyncAnthropic(api_key=anthropic_api_key)
        return AnthropicProviderSession(
            client=client,
            model=model,
            prompt_messages=prompt_messages,
            tools=serialize_anthropic_tools(canonical_tools),
        )
    # Gemini

    if model in GEMINI_MODELS:
        if not gemini_api_key: raise Exception("Gemini API key is missing.")
        client = genai.Client(api_key=gemini_api_key)
        return GeminiProviderSession(
            client=client,
            model=model,
            prompt_messages=prompt_messages,
            tools=serialize_gemini_tools(canonical_tools),
        )
    raise ValueError(f"Unsupported model: {model.value}")

```

The factory first generates **canonical tool definitions** via `canonical_tool_definitions`, then serializes them using provider-specific functions to match each SDK's expected format.

### OpenAI Session Creation Path

When the requested model exists in `OPENAI_MODELS`, the factory validates that `openai_api_key` is present, then instantiates `AsyncOpenAI` from the OpenAI SDK. It constructs an `OpenAIProviderSession` (defined in [`backend/agent/providers/openai.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/openai.py), lines 75-107) by passing the client, model enum, original prompt messages, and tools processed through `serialize_openai_tools`.

### Anthropic Session Creation Path

For models in `ANTHROPIC_MODELS`, the factory verifies `anthropic_api_key` and creates `AsyncAnthropic`. It returns an `AnthropicProviderSession` (located in [`backend/agent/providers/anthropic.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/anthropic.py), lines 27-44) with tools serialized via `serialize_anthropic_tools`. This session handles the translation between OpenAI-style message formats and Claude's native message structure.

### Gemini Session Creation Path

When the model belongs to `GEMINI_MODELS`, the factory checks for `gemini_api_key` and instantiates `genai.Client` from Google's SDK. The resulting `GeminiProviderSession` (found in [`backend/agent/providers/gemini.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/gemini.py), lines 26-45) receives tools processed through `serialize_gemini_tools` and manages Gemini-specific `Content` objects for streaming responses.

## Concrete Provider Session Implementations

Each provider session class implements the streaming and tool-execution logic specific to its underlying SDK:

- **`OpenAIProviderSession`** in [`backend/agent/providers/openai.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/openai.py) manages streaming via the `responses.create` API, handling tool calls and assembling `ProviderTurn` results from OpenAI's event stream.
- **`AnthropicProviderSession`** in [`backend/agent/providers/anthropic.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/anthropic.py) converts the internal prompt format to Claude's message structure, streams Claude's delta responses, and extracts tool calls into the standard turn format.
- **`GeminiProviderSession`** in [`backend/agent/providers/gemini.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/gemini.py) builds Gemini `Content` objects from the prompt messages, streams the model's response chunks, and parses function calls and text completions into the unified `ProviderTurn` interface.

## Practical Implementation Examples

The factory enables seamless provider switching by changing only the model enum and corresponding API key:

```python

# OpenAI GPT-4.1 session

session = create_provider_session(
    model=Llm.GPT_4_1_2025_04_14,
    prompt_messages=messages,
    should_generate_images=True,
    openai_api_key=os.getenv("OPENAI_API_KEY"),
    openai_base_url=None,
    anthropic_api_key=None,
    gemini_api_key=None,
)

```

```python

# Anthropic Claude Opus 4.6 session

session = create_provider_session(
    model=Llm.CLAUDE_OPUS_4_6,
    prompt_messages=messages,
    should_generate_images=False,
    openai_api_key=None,
    openai_base_url=None,
    anthropic_api_key=os.getenv("ANTHROPIC_API_KEY"),
    gemini_api_key=None,
)

```

```python

# Gemini 3 Flash Preview session

session = create_provider_session(
    model=Llm.GEMINI_3_FLASH_PREVIEW_HIGH,
    prompt_messages=messages,
    should_generate_images=False,
    openai_api_key=None,
    openai_base_url=None,
    anthropic_api_key=None,
    gemini_api_key=os.getenv("GEMINI_API_KEY"),
)

```

## Summary

- **Enum-based routing**: The factory uses membership checks against `OPENAI_MODELS`, `ANTHROPIC_MODELS`, and `GEMINI_MODELS` to determine the correct provider without string parsing.
- **Credential isolation**: Each provider path validates its specific API key (`openai_api_key`, `anthropic_api_key`, or `gemini_api_key`) before instantiating the corresponding SDK client.
- **Tool serialization**: Canonical tool definitions are transformed via provider-specific serializers (`serialize_openai_tools`, `serialize_anthropic_tools`, `serialize_gemini_tools`) to match each API's schema requirements.
- **Unified interface**: All factory outputs implement the `ProviderSession` interface, abstracting provider-specific streaming and response handling from the rest of the application.

## Frequently Asked Questions

### How does the factory determine which AI provider to instantiate?

The factory checks if the requested `Llm` enum value exists in the provider-specific sets defined in [`backend/llm.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/llm.py). If the model is in `OPENAI_MODELS`, it creates an `OpenAIProviderSession`; if in `ANTHROPIC_MODELS`, an `AnthropicProviderSession`; if in `GEMINI_MODELS`, a `GeminiProviderSession`.

### What happens if an API key is missing for the requested provider?

The factory raises a specific exception immediately before client instantiation. For example, when routing to an OpenAI model, if `openai_api_key` is `None` or empty, the factory raises `"OpenAI API key is missing."` before attempting to create the `AsyncOpenAI` client.

### Are tool definitions handled consistently across different providers?

The factory first generates **canonical tool definitions** via `canonical_tool_definitions`, then serializes them using provider-specific functions to match each SDK's requirements. While the internal representation is consistent, the serialization step ensures OpenAI, Anthropic, and Gemini each receive tools in their native schema format.

### Can the created sessions handle streaming responses?

Yes, each concrete implementation (`OpenAIProviderSession`, `AnthropicProviderSession`, `GeminiProviderSession`) implements provider-specific streaming logic. The factory ensures that regardless of which provider session is created, the calling code can consume streaming responses and tool calls through the uniform `ProviderSession` interface.