# How to Implement a Custom Evaluation Strategy in Twinkle Eval: A Complete Guide

> Learn to implement a custom evaluation strategy in Twinkle Eval. Follow our guide to create, register, and use your own evaluation methods for advanced AI model assessment.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**To implement a custom evaluation strategy in Twinkle Eval, create a class inheriting from `EvaluationStrategy`, implement the `extract_answer` and `get_strategy_name` methods, register the class with `EvaluationStrategyFactory`, and reference the strategy name in your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) file.**

The `ai-twinkle/eval` repository provides a modular framework for evaluating large language model outputs, separating the concerns of LLM interaction from answer extraction. When you implement a custom evaluation strategy in Twinkle Eval, you can define domain-specific parsing logic—such as extracting multiple-choice selections, parsing JSON outputs, or applying custom regex patterns—while leveraging the existing orchestration and configuration infrastructure.

## Understanding the Evaluation Architecture

Twinkle Eval decouples **LLM inference** from **answer extraction** through a strategy pattern. Understanding these components ensures your custom implementation integrates correctly.

### Core Components

| Component | Responsibility | Key File |
|-----------|----------------|----------|
| **LLM wrapper** | Sends prompts to the chosen model and returns raw completions. | [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) |
| **Evaluator** | Orchestrates dataset loading, rate-limiting, parallel calls, and forwards raw LLM output to the evaluation strategy. | [[`evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/evaluators.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py#L31-L45) |
| **EvaluationStrategy (abstract)** | Defines the contract each strategy must fulfill via `extract_answer` and `get_strategy_name`. | [[`evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/evaluation_strategies.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py#L14-L32) |
| **EvaluationStrategyFactory** | Maintains a registry mapping `strategy_name` to `StrategyClass` and instantiates strategies from configuration. | [[`evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/evaluation_strategies.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py#L63-L95) |
| **ConfigurationManager** | Loads [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml), validates it, and builds the strategy instance via the factory. | [[`config.py`](https://github.com/ai-twinkle/eval/blob/main/config.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py#L74-L89) |

The plug-in point for custom logic is the `EvaluationStrategyFactory` registry. By registering your subclass, you enable the `ConfigurationManager` to instantiate your strategy when it encounters your custom name in the configuration file.

## Step 1: Create the Custom Strategy Class

Create a new Python file for your strategy. Inherit from `EvaluationStrategy` and implement the two required abstract methods.

```python

# my_custom_strategy.py

from typing import Optional, Dict, Any
import re

from twinkle_eval.evaluation_strategies import EvaluationStrategy


class MyCustomStrategy(EvaluationStrategy):
    """
    Extracts answers following the phrase 'The correct option is' 
    and a single letter A-D.
    """

    def __init__(self, config: Optional[Dict[str, Any]] = None):
        super().__init__(config)
        # Allow pattern override via configuration

        default_pattern = r"The correct option is\s*[:\-]?\s*([A-D])"
        self.pattern = (config or {}).get("pattern", default_pattern)

    def get_strategy_name(self) -> str:
        """Return the unique identifier for this strategy."""
        return "my_custom"

    def extract_answer(self, llm_output: str) -> Optional[str]:
        """
        Parse the raw LLM output and return the extracted answer.
        Returns None if extraction fails.
        """
        if not self.validate_output(llm_output):
            return None

        match = re.search(self.pattern, llm_output, flags=re.IGNORECASE)
        return match.group(1).strip().upper() if match else None

```

**Key implementation details:**

- **Inheritance**: Extend `EvaluationStrategy` defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) (lines 14-32).
- **Required methods**: You must implement `get_strategy_name()` and `extract_answer()`.
- **Validation**: Use `self.validate_output()` (provided by the base class) to handle empty or null inputs gracefully.
- **Configuration**: Accept a `config` dictionary in `__init__` to make your strategy reusable without code changes.

## Step 2: Register the Strategy with the Factory

Before the configuration loader can use your strategy, you must register it with `EvaluationStrategyFactory`. Perform this registration in your application entry point before loading the configuration.

```python

# twinkle_eval/main.py (or your entry point script)

from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory
from my_custom_strategy import MyCustomStrategy

# Register the strategy under the name "my_custom"

EvaluationStrategyFactory.register_strategy("my_custom", MyCustomStrategy)

```

The `register_strategy` method updates the internal `_registry` dictionary (see lines 66-71 of [`evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/evaluation_strategies.py)). Once registered, the factory can instantiate your strategy via `create_strategy(name, config)`.

## Step 3: Configure the Strategy in config.yaml

Reference your registered strategy in the configuration file to enable the `ConfigurationManager` to instantiate it automatically.

```yaml

# config.yaml

evaluation:
  evaluation_method: my_custom  # Must match the name used in register_strategy

  strategy_config:
    pattern: "The answer is[:\\s]*([A-D])"  # Optional: override default regex

```

When `ConfigurationManager` loads this file, it calls `EvaluationStrategyFactory.create_strategy` (lines 75-82 of [`config.py`](https://github.com/ai-twinkle/eval/blob/main/config.py)), passing `"my_custom"` and the `strategy_config` dictionary. The factory returns an instance of `MyCustomStrategy` with your custom pattern configured.

## Complete Working Example

The following runnable script demonstrates the entire flow without requiring a live LLM, using a mock response to verify the extraction logic.

```python

# demo_custom_strategy.py

import json
from typing import Dict, Optional

# Import Twinkle Eval core components

from twinkle_eval.evaluation_strategies import EvaluationStrategy, EvaluationStrategyFactory
from twinkle_eval.evaluators import Evaluator
from twinkle_eval.models import LLM

# ----------------------------------------------------------------------

# 1. Define the custom strategy

# ----------------------------------------------------------------------

import re

class MyCustomStrategy(EvaluationStrategy):
    def __init__(self, config: Optional[Dict] = None):
        super().__init__(config)
        default_pattern = r"The correct option is\s*[:\-]?\s*([A-D])"
        self.pattern = (config or {}).get("pattern", default_pattern)

    def get_strategy_name(self) -> str:
        return "my_custom"

    def extract_answer(self, llm_output: str) -> Optional[str]:
        if not self.validate_output(llm_output):
            return None
        match = re.search(self.pattern, llm_output, flags=re.IGNORECASE)
        return match.group(1).upper() if match else None

# ----------------------------------------------------------------------

# 2. Register with the factory

# ----------------------------------------------------------------------

EvaluationStrategyFactory.register_strategy("my_custom", MyCustomStrategy)

# ----------------------------------------------------------------------

# 3. Mock LLM that returns a deterministic response

# ----------------------------------------------------------------------

class EchoLLM(LLM):
    def call(self, prompt: str, language: str = "zh"):
        class MockChoice:
            def __init__(self):
                self.message = type("msg", (), {"content": "The correct option is B."})()
        class MockResp:
            def __init__(self):
                self.choices = [MockChoice()]
                self.usage = type("u", (), {"completion_tokens": 5, "prompt_tokens": 10, "total_tokens": 15})()
        return MockResp()

# ----------------------------------------------------------------------

# 4. Simulate configuration and execution

# ----------------------------------------------------------------------

demo_cfg = {
    "llm_api": {"type": "mock"},
    "model": {},
    "evaluation": {
        "evaluation_method": "my_custom",
        "strategy_config": {"pattern": r"The correct option is\s*[:\-]?\s*([A-D])"},
        "dataset_paths": []
    },
    "environment": {}
}

# Inject mock LLM

demo_cfg["llm_instance"] = EchoLLM()

# Create strategy instance via factory (mirrors ConfigurationManager behavior)

strategy = EvaluationStrategyFactory.create_strategy(
    demo_cfg["evaluation"]["evaluation_method"],
    demo_cfg["evaluation"]["strategy_config"],
)

# Initialize evaluator

evaluator = Evaluator(
    llm=demo_cfg["llm_instance"],
    evaluation_strategy=strategy,
    config=demo_cfg,
)

# Simulate single question evaluation

question_prompt = "What is 2+2?\nA: 3\nB: 4\nC: 5\nD: 6"
llm_response = demo_cfg["llm_instance"].call(question_prompt, "zh")
raw_output = llm_response.choices[0].message.content
predicted = strategy.extract_answer(raw_output)

print(f"Raw LLM output: {raw_output}")
print(f"Extracted answer: {predicted}")
print(f"Correct: {predicted == 'B'}")

```

This demonstration validates that when you implement a custom evaluation strategy in Twinkle Eval, the factory correctly instantiates your class, passes configuration parameters, and invokes your extraction logic during the evaluation pipeline.

## Key Files and Their Roles

When extending the framework, you will interact with these specific files:

| File | Purpose | Source Reference |
|------|---------|------------------|
| [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) | Contains the abstract `EvaluationStrategy` base class, built-in implementations, and the `EvaluationStrategyFactory` registry. | [[`evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/evaluation_strategies.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) |
| [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) | The `Evaluator` class orchestrates dataset iteration and calls `evaluation_strategy.extract_answer` for each LLM response. | [[`evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/evaluators.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) |
| [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) | The `ConfigurationManager` loads YAML configuration and constructs strategy instances via the factory. | [[`config.py`](https://github.com/ai-twinkle/eval/blob/main/config.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) |
| [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) | Recommended location for `EvaluationStrategyFactory.register_strategy` calls before configuration loading. | [[`main.py`](https://github.com/ai-twinkle/eval/blob/main/main.py)](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) |

## Summary

To successfully implement a custom evaluation strategy in Twinkle Eval, follow these essential steps:

- **Inherit from the base class**: Create a subclass of `EvaluationStrategy` defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) and implement the required `extract_answer` and `get_strategy_name` methods.
- **Accept configuration**: Design your `__init__` method to accept an optional `config` dictionary, allowing users to customize behavior via YAML without modifying code.
- **Register with the factory**: Call `EvaluationStrategyFactory.register_strategy` with your strategy name and class in your entry point before the configuration loads.
- **Configure via YAML**: Set `evaluation.evaluation_method` to your registered name and provide `strategy_config` parameters in [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml).
- **Leverage validation**: Use `self.validate_output()` from the base class to handle empty or malformed LLM responses gracefully.

## Frequently Asked Questions

### What methods must I implement when creating a custom evaluation strategy?

You must implement two abstract methods defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py): `get_strategy_name()`, which returns a unique string identifier for your strategy, and `extract_answer(llm_output: str)`, which parses the raw LLM response and returns the extracted answer or `None` if extraction fails. The base class provides `validate_output()` for input sanitization, which you should call at the start of your extraction logic.

### Where should I register my custom evaluation strategy?

Register your strategy in your application's entry point, such as [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) or a dedicated plugin loader, before the `ConfigurationManager` loads the YAML configuration. Call `EvaluationStrategyFactory.register_strategy("your_name", YourStrategyClass)` to add your class to the internal registry. If registration occurs after configuration loading, the factory will raise an error when attempting to create the strategy instance.

### Can I pass configuration parameters to my custom strategy?

Yes, the `EvaluationStrategy` base class accepts an optional `config` dictionary in its `__init__` method. When you define `evaluation.strategy_config` in [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml), the `ConfigurationManager` passes this dictionary to your strategy's constructor via `EvaluationStrategyFactory.create_strategy()`. This allows users to customize regex patterns, parsing rules, or thresholds without modifying your strategy's source code.

### How does the Evaluator use my custom strategy?

The `Evaluator` class in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) orchestrates the evaluation pipeline by loading datasets, managing parallel LLM calls, and processing responses. For each raw LLM output, the evaluator calls `evaluation_strategy.extract_answer()` (as implemented in lines 99-112 of [`evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/evaluators.py)) to obtain the predicted answer. It then compares this prediction against the ground truth to compute accuracy metrics, handling the entire lifecycle independently of your specific extraction logic.