# How to Add Support for a New LLM Provider in the Twinkle Eval Architecture

> Learn how to add a new LLM provider to the Twinkle Eval architecture. Implement a subclass, register it, and extend your configuration schema for seamless integration.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**To add support for a new LLM provider in Twinkle Eval, implement a subclass of the `LLM` abstract base class, register it in the `LLMFactory` registry, and extend your configuration schema to include provider-specific keys.**

Twinkle Eval interacts with language model APIs through a thin abstraction layer defined in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py). This factory-based architecture allows you to integrate new providers—such as Anthropic, Cohere, or local inference endpoints—without modifying downstream evaluation components. Because all consumers reference the abstract `LLM` interface, your new provider becomes a drop-in component once registered.

## Understanding the LLM Abstraction Layer

### The LLM Base Class

The required interface is declared in the abstract `LLM` class located in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) (lines 11-25). Any concrete implementation must provide:

- `validate_config()`: Validates provider-specific configuration keys (e.g., API keys, base URLs)
- `call(question_text, prompt_lang)`: Executes the API request and returns a response object

### The Factory Pattern and Registry

`LLMFactory` maintains a registry mapping provider names to concrete classes. The registry is implemented as a simple dictionary `_registry` defined at module load time in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) (lines 12-15). When the system starts, `ConfigurationManager` reads `config["llm_api"]["type"]` and delegates instantiation to `LLMFactory.create_llm()` (lines 62-66 in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)).

### Configuration-Driven Instantiation

The `ConfigurationManager` in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) applies default values during startup. You can see how defaults are handled in the `_apply_defaults` method (lines 70-73), where the `"type"` key defaults to `"openai"` if not specified.

## Step-by-Step Implementation Guide

### 1. Implement a Subclass of the LLM Base Class

Create a new class in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) that inherits from `LLM`. Refer to the `OpenAIModel` implementation (lines 28-61) as a complete reference. Your subclass must:

- Accept a configuration dictionary in `__init__`
- Implement `validate_config()` to check for required keys
- Implement `call()` to send requests to the provider's API

### 2. Register Your Provider in the LLMFactory Registry

Add an entry to `LLMFactory._registry` mapping a unique string identifier to your class:

```python

# In twinkle_eval/models.py, near the registry definition (lines 12-15)

LLMFactory._registry["your_provider_name"] = YourProviderModel

```

### 3. Extend the Configuration Schema

Add any new required keys under the `llm_api` section of your configuration file (e.g., `your_api_key`, `base_url`). Optionally, provide defaults in `ConfigurationManager._apply_defaults` (lines 70-73 in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)) to ensure backward compatibility.

### 4. Validate with Unit Tests (Optional)

Verify that `LLMFactory.create_llm("your_provider_name", cfg)` returns an instance of your class. Mock the provider's client to test that `call()` correctly handles API responses and error cases.

## Complete Implementation Example: Adding Anthropic Support

The following example demonstrates adding Anthropic Claude support while maintaining compatibility with the existing evaluation pipeline.

First, implement the `AnthropicModel` class in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py):

```python

# twinkle_eval/models.py

from typing import Dict, Any
from .logger import log_error
from anthropic import Anthropic
from openai.types.chat import ChatCompletion  # Maintains return type compatibility

class AnthropicModel(LLM):
    """Anthropic-compatible LLM implementation."""

    def __init__(self, config: Dict[str, Any]):
        super().__init__(config)
        self.validate_config()
        self._initialize_client()

    def validate_config(self) -> bool:
        """Verify required Anthropic configuration keys exist."""
        required = ["api_key", "base_url"]
        for key in required:
            if key not in self.config["llm_api"]:
                raise ValueError(f"Missing required config key: llm_api.{key}")
        return True

    def _initialize_client(self):
        """Initialize the Anthropic client with configuration values."""
        api_cfg = self.config["llm_api"]
        self.client = Anthropic(
            api_key=api_cfg["api_key"],
            base_url=api_cfg.get("base_url", "https://api.anthropic.com"),
        )

    def _build_messages(self, question_text: str, prompt_lang: str) -> list:
        """Construct message list using system prompts from config."""
        eval_cfg = self.config["evaluation"]
        if eval_cfg["evaluation_method"] == "box":
            sys_prompt_cfg = eval_cfg.get("system_prompt", {})
            sys_prompt = (
                sys_prompt_cfg.get(prompt_lang)
                or sys_prompt_cfg.get("zh")
                or ""
            )
            return [
                {"role": "system", "content": sys_prompt},
                {"role": "user", "content": question_text},
            ]
        return [{"role": "user", "content": question_text}]

    def call(self, question_text: str, prompt_lang: str = "zh") -> ChatCompletion:
        """Execute API call and return standardized response."""
        messages = self._build_messages(question_text, prompt_lang)
        model_cfg = self.config["model"]
        try:
            response = self.client.messages.create(
                model=model_cfg["name"],
                max_tokens=model_cfg["max_tokens"],
                temperature=model_cfg["temperature"],
                top_p=model_cfg["top_p"],
                messages=messages,
            )
            # Convert to OpenAI-style ChatCompletion if needed for compatibility

            return response  # type: ignore

        except Exception as e:
            log_error(f"Anthropic API error: {e}")
            raise e

```

Next, register the class in the factory registry:

```python

# In twinkle_eval/models.py, following the registry definition

LLMFactory._registry["anthropic"] = AnthropicModel

```

Finally, update your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) to use the new provider:

```yaml
llm_api:
  type: anthropic          # Selects the registered provider

  api_key: YOUR_ANTHROPIC_KEY
  base_url: https://api.anthropic.com

model:
  name: claude-3-opus-20240229
  max_tokens: 4096
  temperature: 0.0
  top_p: 1.0

```

## Key Files in the Architecture

- **[`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py)**: Defines the `LLM` abstraction, concrete implementations (e.g., `OpenAIModel`), and the `LLMFactory` registry (lines 12-15, 28-61).
- **[`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)**: Loads YAML configuration, applies defaults in `_apply_defaults` (lines 70-73), and creates LLM instances via `LLMFactory.create_llm` (lines 62-66).
- **[`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py)**: Consumes the `llm_instance` from configuration; requires no changes when adding new providers.
- **[`setup.py`](https://github.com/ai-twinkle/eval/blob/main/setup.py) / [`requirements.txt`](https://github.com/ai-twinkle/eval/blob/main/requirements.txt)**: Add the new provider's Python package (e.g., `anthropic`) to ensure the code can be imported.

## Summary

- **Implement the interface**: Create a subclass of `LLM` in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) with `validate_config` and `call` methods, using `OpenAIModel` (lines 28-61) as a reference.
- **Register the provider**: Add your class to `LLMFactory._registry` with a unique string key (lines 12-15).
- **Configure the integration**: Extend the `llm_api` configuration section with provider-specific parameters and update defaults in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) (lines 70-73) if necessary.
- **Zero downstream changes**: Consumers in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) and other modules reference only the abstract `LLM` interface, so no other code requires modification.

## Frequently Asked Questions

### Do I need to modify the evaluation logic to support a new LLM provider?

No. Because the architecture uses an abstract `LLM` base class and the `LLMFactory` pattern, downstream components in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) interact only with the abstract interface. Once you register your new provider and set the `type` key in your configuration, the factory handles instantiation automatically without requiring changes to evaluation logic.

### What methods must I implement when subclassing the LLM base class?

You must implement `validate_config()` to check for required configuration keys (such as API keys or base URLs) and `call(question_text, prompt_lang)` to execute the actual API request. The `__init__` method should call `validate_config()` and initialize any provider-specific clients. Refer to the `LLM` abstract class definition in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) (lines 11-25) for the exact interface contract.

### How does Twinkle Eval handle provider-specific configuration parameters?

The `ConfigurationManager` in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) loads the entire `llm_api` dictionary from your configuration file and passes it to your implementation's `__init__` method. You define which keys are required in your `validate_config()` method. For optional parameters with default values, add them to `ConfigurationManager._apply_defaults` (lines 70-73) to ensure the configuration is always complete.

### Can I support multiple LLM providers simultaneously in the same evaluation run?

The current architecture instantiates a single LLM provider per configuration session based on the `config["llm_api"]["type"]` value. To evaluate against multiple providers simultaneously, you would need to run separate evaluation processes with different configuration files, or extend the factory to support a composite LLM implementation that round-robins or compares multiple providers within a single `call()` invocation.