How EvaluationStrategyFactory Registers and Manages Evaluation Strategies in ai-twinkle/eval

The EvaluationStrategyFactory acts as a central registry that decouples strategy instantiation from application logic by mapping string identifiers to concrete strategy classes, supporting both static built-in registrations and dynamic runtime extensions.

The ai-twinkle/eval repository provides a flexible framework for evaluating LLM outputs through pluggable evaluation strategies. At the core of this extensibility lies the EvaluationStrategyFactory, which enables the rest of the codebase to request strategies by name without hardcoding class references or instantiation logic.

Understanding the EvaluationStrategyFactory Architecture

The EvaluationStrategyFactory is implemented in twinkle_eval/evaluation_strategies.py as a factory class that manages the lifecycle of evaluation strategies. It maintains a class-level dictionary _registry that stores mappings between string identifiers and strategy classes, allowing the factory to instantiate appropriate strategies at runtime while keeping calling code simple and version-agnostic.

How Evaluation Strategies Are Registered

Registration occurs through two distinct mechanisms: static definition for built-in strategies and dynamic registration for custom extensions.

Built-in Strategy Registration

Built-in strategies are registered statically within the class body. The _registry dictionary maps short string identifiers to their corresponding strategy classes at lines 66-70 of twinkle_eval/evaluation_strategies.py:

_registry: Dict[str, Type[EvaluationStrategy]] = {
    "pattern": PatternMatchingStrategy,
    "box": BoxExtractionStrategy,
    "custom_regex": CustomRegexStrategy,
}

This static mapping ensures that standard strategies like PatternMatchingStrategy (identifier "pattern"), BoxExtractionStrategy (identifier "box"), and CustomRegexStrategy (identifier "custom_regex") are available immediately when the factory loads.

Dynamic Strategy Registration

The factory supports runtime extensibility through the register_strategy() class method defined at lines 73-76. This allows external code to add new strategies without modifying the factory's source code:

@classmethod
def register_strategy(cls, name: str, strategy_class: Type[EvaluationStrategy]):
    """Register a new evaluation strategy."""
    cls._registry[name] = strategy_class

When called, this method simply inserts the new mapping into the _registry dictionary, making the strategy immediately available to create_strategy() and get_available_types().

Instantiating Strategies with the Factory

The primary consumer interface is the create_strategy() method, which handles instantiation, validation, and error handling. Implemented at lines 78-88 of twinkle_eval/evaluation_strategies.py, this method checks the registry, raises descriptive errors for unknown types, and returns initialized instances:

@classmethod
def create_strategy(
    cls, strategy_type: str, config: Optional[Dict[str, Any]] = None
) -> EvaluationStrategy:
    if strategy_type not in cls._registry:
        available = ", ".join(cls._registry.keys())
        raise ValueError(
            f"Unsupported strategy type: {strategy_type}. Available types: {available}"
        )
    strategy_class = cls._registry[strategy_type]
    return strategy_class(config)

The method passes optional configuration dictionaries directly to strategy constructors, enabling runtime parameterization without factory modifications.

Practical Examples

Using a Built-in Strategy

To instantiate the pattern matching strategy with custom regex patterns:

from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

# Request the "pattern" strategy (PatternMatchingStrategy) with optional config

strategy = EvaluationStrategyFactory.create_strategy(
    "pattern", config={"patterns": [r"答案是\s?([A-D])"]}
)

answer = strategy.extract_answer(llm_output_text)

The factory resolves "pattern" to PatternMatchingStrategy and initializes it with the provided configuration.

Registering a Custom Strategy

Extend the factory with a custom implementation without modifying source code:

from typing import Optional
from twinkle_eval.evaluation_strategies import (
    EvaluationStrategy,
    EvaluationStrategyFactory,
)

class FirstLetterStrategy(EvaluationStrategy):
    """Extract the first alphabetic character from output."""
    def extract_answer(self, llm_output: str) -> Optional[str]:
        for ch in llm_output:
            if ch.isalpha():
                return ch.upper()
        return None

    def get_strategy_name(self) -> str:
        return "first_letter"

# Register dynamically

EvaluationStrategyFactory.register_strategy("first_letter", FirstLetterStrategy)

# Use immediately

strategy = EvaluationStrategyFactory.create_strategy("first_letter")
print(strategy.extract_answer("The answer is B."))   # → "T"

Listing Available Strategies

The factory provides introspection capabilities used by the CLI in twinkle_eval/cli.py:

from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

available = EvaluationStrategyFactory.get_available_types()
print(available)

# Output: ['pattern', 'box', 'custom_regex', 'first_letter']

Summary

• The EvaluationStrategyFactory in twinkle_eval/evaluation_strategies.py acts as a central registry that decouples strategy instantiation from application logic using the factory pattern.
• Built-in strategies are statically registered in the _registry dictionary at class definition time, mapping identifiers like "pattern", "box", and "custom_regex" to their respective classes.
• Dynamic registration is supported via register_strategy(), allowing external modules to add custom strategies at runtime without modifying the factory source code.
• The create_strategy() method handles instantiation, validation, and error handling, passing configuration dictionaries directly to strategy constructors.
• Introspection methods like get_available_types() enable CLI tools and configuration loaders to discover available strategies dynamically.

Frequently Asked Questions

What is the difference between EvaluationStrategyFactory and individual strategy classes?

The EvaluationStrategyFactory is a creational design pattern implementation that manages the lifecycle and discovery of strategy objects, while individual strategy classes like PatternMatchingStrategy or BoxExtractionStrategy contain the specific implementation logic for extracting or evaluating answers. The factory decouples the rest of the application from knowing which specific class to instantiate, allowing strategies to be referenced by simple string identifiers and swapped without changing calling code.

Can I register a custom strategy without modifying the source code?

Yes, the factory supports runtime registration through the register_strategy() class method. You can define a custom class inheriting from EvaluationStrategy and register it by calling EvaluationStrategyFactory.register_strategy("my_strategy", MyStrategyClass). Once registered, the strategy becomes immediately available to create_strategy() and get_available_types() throughout the application without requiring any changes to twinkle_eval/evaluation_strategies.py.

How does the factory handle invalid strategy type requests?

When create_strategy() receives a strategy type that is not present in the _registry dictionary, it raises a ValueError with a descriptive message listing all available strategy types. This validation occurs at lines 80-85 of twinkle_eval/evaluation_strategies.py, ensuring that configuration errors are caught during instantiation with helpful debugging information rather than causing obscure import errors or returning None values.

Where is EvaluationStrategyFactory used in the ai-twinkle/eval codebase?

The factory is utilized across multiple modules: twinkle_eval/config.py uses it to instantiate strategies based on user configuration files; twinkle_eval/cli.py calls get_available_types() to display valid options in command-line help messages; and twinkle_eval/main.py orchestrates full evaluation runs using strategies created by the factory. This widespread usage demonstrates its role as the central dependency injection point for evaluation logic in the repository.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}