How to Add Support for a New LLM Provider in the Twinkle Eval Architecture

To add support for a new LLM provider in Twinkle Eval, implement a subclass of the LLM abstract base class, register it in the LLMFactory registry, and extend your configuration schema to include provider-specific keys.

Twinkle Eval interacts with language model APIs through a thin abstraction layer defined in twinkle_eval/models.py. This factory-based architecture allows you to integrate new providers—such as Anthropic, Cohere, or local inference endpoints—without modifying downstream evaluation components. Because all consumers reference the abstract LLM interface, your new provider becomes a drop-in component once registered.

Understanding the LLM Abstraction Layer

The LLM Base Class

The required interface is declared in the abstract LLM class located in twinkle_eval/models.py (lines 11-25). Any concrete implementation must provide:

• validate_config(): Validates provider-specific configuration keys (e.g., API keys, base URLs)
• call(question_text, prompt_lang): Executes the API request and returns a response object

The Factory Pattern and Registry

LLMFactory maintains a registry mapping provider names to concrete classes. The registry is implemented as a simple dictionary _registry defined at module load time in twinkle_eval/models.py (lines 12-15). When the system starts, ConfigurationManager reads config["llm_api"]["type"] and delegates instantiation to LLMFactory.create_llm() (lines 62-66 in twinkle_eval/config.py).

Configuration-Driven Instantiation

The ConfigurationManager in twinkle_eval/config.py applies default values during startup. You can see how defaults are handled in the _apply_defaults method (lines 70-73), where the "type" key defaults to "openai" if not specified.

Step-by-Step Implementation Guide

1. Implement a Subclass of the LLM Base Class

Create a new class in twinkle_eval/models.py that inherits from LLM. Refer to the OpenAIModel implementation (lines 28-61) as a complete reference. Your subclass must:

• Accept a configuration dictionary in __init__
• Implement validate_config() to check for required keys
• Implement call() to send requests to the provider's API

2. Register Your Provider in the LLMFactory Registry

Add an entry to LLMFactory._registry mapping a unique string identifier to your class:

# In twinkle_eval/models.py, near the registry definition (lines 12-15)

LLMFactory._registry["your_provider_name"] = YourProviderModel

3. Extend the Configuration Schema

Add any new required keys under the llm_api section of your configuration file (e.g., your_api_key, base_url). Optionally, provide defaults in ConfigurationManager._apply_defaults (lines 70-73 in twinkle_eval/config.py) to ensure backward compatibility.

4. Validate with Unit Tests (Optional)

Verify that LLMFactory.create_llm("your_provider_name", cfg) returns an instance of your class. Mock the provider's client to test that call() correctly handles API responses and error cases.

Complete Implementation Example: Adding Anthropic Support

The following example demonstrates adding Anthropic Claude support while maintaining compatibility with the existing evaluation pipeline.

First, implement the AnthropicModel class in twinkle_eval/models.py:

# twinkle_eval/models.py

from typing import Dict, Any
from .logger import log_error
from anthropic import Anthropic
from openai.types.chat import ChatCompletion  # Maintains return type compatibility

class AnthropicModel(LLM):
    """Anthropic-compatible LLM implementation."""

    def __init__(self, config: Dict[str, Any]):
        super().__init__(config)
        self.validate_config()
        self._initialize_client()

    def validate_config(self) -> bool:
        """Verify required Anthropic configuration keys exist."""
        required = ["api_key", "base_url"]
        for key in required:
            if key not in self.config["llm_api"]:
                raise ValueError(f"Missing required config key: llm_api.{key}")
        return True

    def _initialize_client(self):
        """Initialize the Anthropic client with configuration values."""
        api_cfg = self.config["llm_api"]
        self.client = Anthropic(
            api_key=api_cfg["api_key"],
            base_url=api_cfg.get("base_url", "https://api.anthropic.com"),
        )

    def _build_messages(self, question_text: str, prompt_lang: str) -> list:
        """Construct message list using system prompts from config."""
        eval_cfg = self.config["evaluation"]
        if eval_cfg["evaluation_method"] == "box":
            sys_prompt_cfg = eval_cfg.get("system_prompt", {})
            sys_prompt = (
                sys_prompt_cfg.get(prompt_lang)
                or sys_prompt_cfg.get("zh")
                or ""
            )
            return [
                {"role": "system", "content": sys_prompt},
                {"role": "user", "content": question_text},
            ]
        return [{"role": "user", "content": question_text}]

    def call(self, question_text: str, prompt_lang: str = "zh") -> ChatCompletion:
        """Execute API call and return standardized response."""
        messages = self._build_messages(question_text, prompt_lang)
        model_cfg = self.config["model"]
        try:
            response = self.client.messages.create(
                model=model_cfg["name"],
                max_tokens=model_cfg["max_tokens"],
                temperature=model_cfg["temperature"],
                top_p=model_cfg["top_p"],
                messages=messages,
            )
            # Convert to OpenAI-style ChatCompletion if needed for compatibility

            return response  # type: ignore

        except Exception as e:
            log_error(f"Anthropic API error: {e}")
            raise e

Next, register the class in the factory registry:

# In twinkle_eval/models.py, following the registry definition

LLMFactory._registry["anthropic"] = AnthropicModel

Finally, update your config.yaml to use the new provider:

llm_api:
  type: anthropic          # Selects the registered provider

  api_key: YOUR_ANTHROPIC_KEY
  base_url: https://api.anthropic.com

model:
  name: claude-3-opus-20240229
  max_tokens: 4096
  temperature: 0.0
  top_p: 1.0

Key Files in the Architecture

• twinkle_eval/models.py: Defines the LLM abstraction, concrete implementations (e.g., OpenAIModel), and the LLMFactory registry (lines 12-15, 28-61).
• twinkle_eval/config.py: Loads YAML configuration, applies defaults in _apply_defaults (lines 70-73), and creates LLM instances via LLMFactory.create_llm (lines 62-66).
• twinkle_eval/evaluators.py: Consumes the llm_instance from configuration; requires no changes when adding new providers.
• setup.py / requirements.txt: Add the new provider's Python package (e.g., anthropic) to ensure the code can be imported.

Summary

• Implement the interface: Create a subclass of LLM in twinkle_eval/models.py with validate_config and call methods, using OpenAIModel (lines 28-61) as a reference.
• Register the provider: Add your class to LLMFactory._registry with a unique string key (lines 12-15).
• Configure the integration: Extend the llm_api configuration section with provider-specific parameters and update defaults in twinkle_eval/config.py (lines 70-73) if necessary.
• Zero downstream changes: Consumers in twinkle_eval/evaluators.py and other modules reference only the abstract LLM interface, so no other code requires modification.

Frequently Asked Questions

Do I need to modify the evaluation logic to support a new LLM provider?

No. Because the architecture uses an abstract LLM base class and the LLMFactory pattern, downstream components in twinkle_eval/evaluators.py interact only with the abstract interface. Once you register your new provider and set the type key in your configuration, the factory handles instantiation automatically without requiring changes to evaluation logic.

What methods must I implement when subclassing the LLM base class?

You must implement validate_config() to check for required configuration keys (such as API keys or base URLs) and call(question_text, prompt_lang) to execute the actual API request. The __init__ method should call validate_config() and initialize any provider-specific clients. Refer to the LLM abstract class definition in twinkle_eval/models.py (lines 11-25) for the exact interface contract.

How does Twinkle Eval handle provider-specific configuration parameters?

The ConfigurationManager in twinkle_eval/config.py loads the entire llm_api dictionary from your configuration file and passes it to your implementation's __init__ method. You define which keys are required in your validate_config() method. For optional parameters with default values, add them to ConfigurationManager._apply_defaults (lines 70-73) to ensure the configuration is always complete.

Can I support multiple LLM providers simultaneously in the same evaluation run?

The current architecture instantiates a single LLM provider per configuration session based on the config["llm_api"]["type"] value. To evaluate against multiple providers simultaneously, you would need to run separate evaluation processes with different configuration files, or extend the factory to support a composite LLM implementation that round-robins or compares multiple providers within a single call() invocation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}