# How EvaluationStrategyFactory Registers and Manages Evaluation Strategies in ai-twinkle/eval

> Discover how the EvaluationStrategyFactory in ai-twinkle/eval registers and manages evaluation strategies. Learn about dynamic registration and decoupling strategy instantiation.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: internals
- Published: 2026-02-23

---

**The `EvaluationStrategyFactory` acts as a central registry that decouples strategy instantiation from application logic by mapping string identifiers to concrete strategy classes, supporting both static built-in registrations and dynamic runtime extensions.**

The `ai-twinkle/eval` repository provides a flexible framework for evaluating LLM outputs through pluggable evaluation strategies. At the core of this extensibility lies the `EvaluationStrategyFactory`, which enables the rest of the codebase to request strategies by name without hardcoding class references or instantiation logic.

## Understanding the EvaluationStrategyFactory Architecture

The `EvaluationStrategyFactory` is implemented in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) as a factory class that manages the lifecycle of evaluation strategies. It maintains a class-level dictionary `_registry` that stores mappings between string identifiers and strategy classes, allowing the factory to instantiate appropriate strategies at runtime while keeping calling code simple and version-agnostic.

## How Evaluation Strategies Are Registered

Registration occurs through two distinct mechanisms: static definition for built-in strategies and dynamic registration for custom extensions.

### Built-in Strategy Registration

Built-in strategies are registered statically within the class body. The `_registry` dictionary maps short string identifiers to their corresponding strategy classes at lines 66-70 of [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py):

```python
_registry: Dict[str, Type[EvaluationStrategy]] = {
    "pattern": PatternMatchingStrategy,
    "box": BoxExtractionStrategy,
    "custom_regex": CustomRegexStrategy,
}

```

This static mapping ensures that standard strategies like `PatternMatchingStrategy` (identifier `"pattern"`), `BoxExtractionStrategy` (identifier `"box"`), and `CustomRegexStrategy` (identifier `"custom_regex"`) are available immediately when the factory loads.

### Dynamic Strategy Registration

The factory supports runtime extensibility through the `register_strategy()` class method defined at lines 73-76. This allows external code to add new strategies without modifying the factory's source code:

```python
@classmethod
def register_strategy(cls, name: str, strategy_class: Type[EvaluationStrategy]):
    """Register a new evaluation strategy."""
    cls._registry[name] = strategy_class

```

When called, this method simply inserts the new mapping into the `_registry` dictionary, making the strategy immediately available to `create_strategy()` and `get_available_types()`.

## Instantiating Strategies with the Factory

The primary consumer interface is the `create_strategy()` method, which handles instantiation, validation, and error handling. Implemented at lines 78-88 of [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py), this method checks the registry, raises descriptive errors for unknown types, and returns initialized instances:

```python
@classmethod
def create_strategy(
    cls, strategy_type: str, config: Optional[Dict[str, Any]] = None
) -> EvaluationStrategy:
    if strategy_type not in cls._registry:
        available = ", ".join(cls._registry.keys())
        raise ValueError(
            f"Unsupported strategy type: {strategy_type}. Available types: {available}"
        )
    strategy_class = cls._registry[strategy_type]
    return strategy_class(config)

```

The method passes optional configuration dictionaries directly to strategy constructors, enabling runtime parameterization without factory modifications.

## Practical Examples

### Using a Built-in Strategy

To instantiate the pattern matching strategy with custom regex patterns:

```python
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

# Request the "pattern" strategy (PatternMatchingStrategy) with optional config

strategy = EvaluationStrategyFactory.create_strategy(
    "pattern", config={"patterns": [r"答案是\s?([A-D])"]}
)

answer = strategy.extract_answer(llm_output_text)

```

The factory resolves `"pattern"` to `PatternMatchingStrategy` and initializes it with the provided configuration.

### Registering a Custom Strategy

Extend the factory with a custom implementation without modifying source code:

```python
from typing import Optional
from twinkle_eval.evaluation_strategies import (
    EvaluationStrategy,
    EvaluationStrategyFactory,
)

class FirstLetterStrategy(EvaluationStrategy):
    """Extract the first alphabetic character from output."""
    def extract_answer(self, llm_output: str) -> Optional[str]:
        for ch in llm_output:
            if ch.isalpha():
                return ch.upper()
        return None

    def get_strategy_name(self) -> str:
        return "first_letter"

# Register dynamically

EvaluationStrategyFactory.register_strategy("first_letter", FirstLetterStrategy)

# Use immediately

strategy = EvaluationStrategyFactory.create_strategy("first_letter")
print(strategy.extract_answer("The answer is B."))   # → "T"

```

### Listing Available Strategies

The factory provides introspection capabilities used by the CLI in [`twinkle_eval/cli.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/cli.py):

```python
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

available = EvaluationStrategyFactory.get_available_types()
print(available)

# Output: ['pattern', 'box', 'custom_regex', 'first_letter']

```

## Summary

- The **EvaluationStrategyFactory** in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) acts as a central registry that decouples strategy instantiation from application logic using the factory pattern.
- **Built-in strategies** are statically registered in the `_registry` dictionary at class definition time, mapping identifiers like `"pattern"`, `"box"`, and `"custom_regex"` to their respective classes.
- **Dynamic registration** is supported via `register_strategy()`, allowing external modules to add custom strategies at runtime without modifying the factory source code.
- The `create_strategy()` method handles instantiation, validation, and error handling, passing configuration dictionaries directly to strategy constructors.
- **Introspection methods** like `get_available_types()` enable CLI tools and configuration loaders to discover available strategies dynamically.

## Frequently Asked Questions

### What is the difference between EvaluationStrategyFactory and individual strategy classes?

The **EvaluationStrategyFactory** is a creational design pattern implementation that manages the lifecycle and discovery of strategy objects, while individual strategy classes like `PatternMatchingStrategy` or `BoxExtractionStrategy` contain the specific implementation logic for extracting or evaluating answers. The factory decouples the rest of the application from knowing which specific class to instantiate, allowing strategies to be referenced by simple string identifiers and swapped without changing calling code.

### Can I register a custom strategy without modifying the source code?

Yes, the factory supports **runtime registration** through the `register_strategy()` class method. You can define a custom class inheriting from `EvaluationStrategy` and register it by calling `EvaluationStrategyFactory.register_strategy("my_strategy", MyStrategyClass)`. Once registered, the strategy becomes immediately available to `create_strategy()` and `get_available_types()` throughout the application without requiring any changes to [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py).

### How does the factory handle invalid strategy type requests?

When `create_strategy()` receives a strategy type that is not present in the `_registry` dictionary, it raises a `ValueError` with a descriptive message listing all available strategy types. This validation occurs at lines 80-85 of [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py), ensuring that configuration errors are caught during instantiation with helpful debugging information rather than causing obscure import errors or returning None values.

### Where is EvaluationStrategyFactory used in the ai-twinkle/eval codebase?

The factory is utilized across multiple modules: [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) uses it to instantiate strategies based on user configuration files; [`twinkle_eval/cli.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/cli.py) calls `get_available_types()` to display valid options in command-line help messages; and [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) orchestrates full evaluation runs using strategies created by the factory. This widespread usage demonstrates its role as the central dependency injection point for evaluation logic in the repository.