How to Implement a Custom Evaluation Strategy in Twinkle Eval: A Complete Guide

To implement a custom evaluation strategy in Twinkle Eval, create a class inheriting from EvaluationStrategy, implement the extract_answer and get_strategy_name methods, register the class with EvaluationStrategyFactory, and reference the strategy name in your config.yaml file.

The ai-twinkle/eval repository provides a modular framework for evaluating large language model outputs, separating the concerns of LLM interaction from answer extraction. When you implement a custom evaluation strategy in Twinkle Eval, you can define domain-specific parsing logic—such as extracting multiple-choice selections, parsing JSON outputs, or applying custom regex patterns—while leveraging the existing orchestration and configuration infrastructure.

Understanding the Evaluation Architecture

Twinkle Eval decouples LLM inference from answer extraction through a strategy pattern. Understanding these components ensures your custom implementation integrates correctly.

Core Components

Component	Responsibility	Key File
LLM wrapper	Sends prompts to the chosen model and returns raw completions.	twinkle_eval/models.py
Evaluator	Orchestrates dataset loading, rate-limiting, parallel calls, and forwards raw LLM output to the evaluation strategy.	[evaluators.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py#L31-L45)
EvaluationStrategy (abstract)	Defines the contract each strategy must fulfill via extract_answer and get_strategy_name.	[evaluation_strategies.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py#L14-L32)
EvaluationStrategyFactory	Maintains a registry mapping strategy_name to StrategyClass and instantiates strategies from configuration.	[evaluation_strategies.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py#L63-L95)
ConfigurationManager	Loads config.yaml, validates it, and builds the strategy instance via the factory.	[config.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py#L74-L89)

The plug-in point for custom logic is the EvaluationStrategyFactory registry. By registering your subclass, you enable the ConfigurationManager to instantiate your strategy when it encounters your custom name in the configuration file.

Step 1: Create the Custom Strategy Class

Create a new Python file for your strategy. Inherit from EvaluationStrategy and implement the two required abstract methods.

# my_custom_strategy.py

from typing import Optional, Dict, Any
import re

from twinkle_eval.evaluation_strategies import EvaluationStrategy


class MyCustomStrategy(EvaluationStrategy):
    """
    Extracts answers following the phrase 'The correct option is' 
    and a single letter A-D.
    """

    def __init__(self, config: Optional[Dict[str, Any]] = None):
        super().__init__(config)
        # Allow pattern override via configuration

        default_pattern = r"The correct option is\s*[:\-]?\s*([A-D])"
        self.pattern = (config or {}).get("pattern", default_pattern)

    def get_strategy_name(self) -> str:
        """Return the unique identifier for this strategy."""
        return "my_custom"

    def extract_answer(self, llm_output: str) -> Optional[str]:
        """
        Parse the raw LLM output and return the extracted answer.
        Returns None if extraction fails.
        """
        if not self.validate_output(llm_output):
            return None

        match = re.search(self.pattern, llm_output, flags=re.IGNORECASE)
        return match.group(1).strip().upper() if match else None

Key implementation details:

• Inheritance: Extend EvaluationStrategy defined in twinkle_eval/evaluation_strategies.py (lines 14-32).
• Required methods: You must implement get_strategy_name() and extract_answer().
• Validation: Use self.validate_output() (provided by the base class) to handle empty or null inputs gracefully.
• Configuration: Accept a config dictionary in __init__ to make your strategy reusable without code changes.

Step 2: Register the Strategy with the Factory

Before the configuration loader can use your strategy, you must register it with EvaluationStrategyFactory. Perform this registration in your application entry point before loading the configuration.

# twinkle_eval/main.py (or your entry point script)

from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory
from my_custom_strategy import MyCustomStrategy

# Register the strategy under the name "my_custom"

EvaluationStrategyFactory.register_strategy("my_custom", MyCustomStrategy)

The register_strategy method updates the internal _registry dictionary (see lines 66-71 of evaluation_strategies.py). Once registered, the factory can instantiate your strategy via create_strategy(name, config).

Step 3: Configure the Strategy in config.yaml

Reference your registered strategy in the configuration file to enable the ConfigurationManager to instantiate it automatically.

# config.yaml

evaluation:
  evaluation_method: my_custom  # Must match the name used in register_strategy

  strategy_config:
    pattern: "The answer is[:\\s]*([A-D])"  # Optional: override default regex

When ConfigurationManager loads this file, it calls EvaluationStrategyFactory.create_strategy (lines 75-82 of config.py), passing "my_custom" and the strategy_config dictionary. The factory returns an instance of MyCustomStrategy with your custom pattern configured.

Complete Working Example

The following runnable script demonstrates the entire flow without requiring a live LLM, using a mock response to verify the extraction logic.

# demo_custom_strategy.py

import json
from typing import Dict, Optional

# Import Twinkle Eval core components

from twinkle_eval.evaluation_strategies import EvaluationStrategy, EvaluationStrategyFactory
from twinkle_eval.evaluators import Evaluator
from twinkle_eval.models import LLM

# ----------------------------------------------------------------------

# 1. Define the custom strategy

# ----------------------------------------------------------------------

import re

class MyCustomStrategy(EvaluationStrategy):
    def __init__(self, config: Optional[Dict] = None):
        super().__init__(config)
        default_pattern = r"The correct option is\s*[:\-]?\s*([A-D])"
        self.pattern = (config or {}).get("pattern", default_pattern)

    def get_strategy_name(self) -> str:
        return "my_custom"

    def extract_answer(self, llm_output: str) -> Optional[str]:
        if not self.validate_output(llm_output):
            return None
        match = re.search(self.pattern, llm_output, flags=re.IGNORECASE)
        return match.group(1).upper() if match else None

# ----------------------------------------------------------------------

# 2. Register with the factory

# ----------------------------------------------------------------------

EvaluationStrategyFactory.register_strategy("my_custom", MyCustomStrategy)

# ----------------------------------------------------------------------

# 3. Mock LLM that returns a deterministic response

# ----------------------------------------------------------------------

class EchoLLM(LLM):
    def call(self, prompt: str, language: str = "zh"):
        class MockChoice:
            def __init__(self):
                self.message = type("msg", (), {"content": "The correct option is B."})()
        class MockResp:
            def __init__(self):
                self.choices = [MockChoice()]
                self.usage = type("u", (), {"completion_tokens": 5, "prompt_tokens": 10, "total_tokens": 15})()
        return MockResp()

# ----------------------------------------------------------------------

# 4. Simulate configuration and execution

# ----------------------------------------------------------------------

demo_cfg = {
    "llm_api": {"type": "mock"},
    "model": {},
    "evaluation": {
        "evaluation_method": "my_custom",
        "strategy_config": {"pattern": r"The correct option is\s*[:\-]?\s*([A-D])"},
        "dataset_paths": []
    },
    "environment": {}
}

# Inject mock LLM

demo_cfg["llm_instance"] = EchoLLM()

# Create strategy instance via factory (mirrors ConfigurationManager behavior)

strategy = EvaluationStrategyFactory.create_strategy(
    demo_cfg["evaluation"]["evaluation_method"],
    demo_cfg["evaluation"]["strategy_config"],
)

# Initialize evaluator

evaluator = Evaluator(
    llm=demo_cfg["llm_instance"],
    evaluation_strategy=strategy,
    config=demo_cfg,
)

# Simulate single question evaluation

question_prompt = "What is 2+2?\nA: 3\nB: 4\nC: 5\nD: 6"
llm_response = demo_cfg["llm_instance"].call(question_prompt, "zh")
raw_output = llm_response.choices[0].message.content
predicted = strategy.extract_answer(raw_output)

print(f"Raw LLM output: {raw_output}")
print(f"Extracted answer: {predicted}")
print(f"Correct: {predicted == 'B'}")

This demonstration validates that when you implement a custom evaluation strategy in Twinkle Eval, the factory correctly instantiates your class, passes configuration parameters, and invokes your extraction logic during the evaluation pipeline.

Key Files and Their Roles

When extending the framework, you will interact with these specific files:

File	Purpose	Source Reference
twinkle_eval/evaluation_strategies.py	Contains the abstract EvaluationStrategy base class, built-in implementations, and the EvaluationStrategyFactory registry.	[evaluation_strategies.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py)
twinkle_eval/evaluators.py	The Evaluator class orchestrates dataset iteration and calls evaluation_strategy.extract_answer for each LLM response.	[evaluators.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py)
twinkle_eval/config.py	The ConfigurationManager loads YAML configuration and constructs strategy instances via the factory.	[config.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)
twinkle_eval/main.py	Recommended location for EvaluationStrategyFactory.register_strategy calls before configuration loading.	[main.py](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py)

Summary

To successfully implement a custom evaluation strategy in Twinkle Eval, follow these essential steps:

• Inherit from the base class: Create a subclass of EvaluationStrategy defined in twinkle_eval/evaluation_strategies.py and implement the required extract_answer and get_strategy_name methods.
• Accept configuration: Design your __init__ method to accept an optional config dictionary, allowing users to customize behavior via YAML without modifying code.
• Register with the factory: Call EvaluationStrategyFactory.register_strategy with your strategy name and class in your entry point before the configuration loads.
• Configure via YAML: Set evaluation.evaluation_method to your registered name and provide strategy_config parameters in config.yaml.
• Leverage validation: Use self.validate_output() from the base class to handle empty or malformed LLM responses gracefully.

Frequently Asked Questions

What methods must I implement when creating a custom evaluation strategy?

You must implement two abstract methods defined in twinkle_eval/evaluation_strategies.py: get_strategy_name(), which returns a unique string identifier for your strategy, and extract_answer(llm_output: str), which parses the raw LLM response and returns the extracted answer or None if extraction fails. The base class provides validate_output() for input sanitization, which you should call at the start of your extraction logic.

Where should I register my custom evaluation strategy?

Register your strategy in your application's entry point, such as twinkle_eval/main.py or a dedicated plugin loader, before the ConfigurationManager loads the YAML configuration. Call EvaluationStrategyFactory.register_strategy("your_name", YourStrategyClass) to add your class to the internal registry. If registration occurs after configuration loading, the factory will raise an error when attempting to create the strategy instance.

Can I pass configuration parameters to my custom strategy?

Yes, the EvaluationStrategy base class accepts an optional config dictionary in its __init__ method. When you define evaluation.strategy_config in config.yaml, the ConfigurationManager passes this dictionary to your strategy's constructor via EvaluationStrategyFactory.create_strategy(). This allows users to customize regex patterns, parsing rules, or thresholds without modifying your strategy's source code.

How does the Evaluator use my custom strategy?

The Evaluator class in twinkle_eval/evaluators.py orchestrates the evaluation pipeline by loading datasets, managing parallel LLM calls, and processing responses. For each raw LLM output, the evaluator calls evaluation_strategy.extract_answer() (as implemented in lines 99-112 of evaluators.py) to obtain the predicted answer. It then compares this prediction against the ground truth to compute accuracy metrics, handling the entire lifecycle independently of your specific extraction logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}