# How Twinkle Eval Handles Answer Extraction from Model Responses

> Discover how Twinkle Eval extracts answers from model responses using regex patterns, LaTeX detection, or custom rules. Optimize your LLM evaluations.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**Twinkle Eval delegates answer extraction to pluggable evaluation strategies that parse LLM outputs using regex patterns, LaTeX box detection, or custom user-defined rules.**

Twinkle Eval, an open-source evaluation framework in the `ai-twinkle/eval` repository, isolates final answers from raw LLM completions through a strategy-based architecture. This design decouples the evaluation orchestration from the parsing logic, allowing users to handle diverse output formats ranging from multiple-choice letters to boxed mathematical expressions.

## Overview of the Answer Extraction Pipeline

The extraction workflow centers on the **`Evaluator`** class in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py), which coordinates between the LLM interface and the parsing layer. When processing a response, the evaluator invokes a configurable **`EvaluationStrategy`** to isolate the answer token.

The pipeline follows four distinct stages:

1. **LLM Invocation** – The `Evaluator` transmits a formatted prompt and receives the raw `content` string.
2. **Strategy Resolution** – At initialization, `Config` (lines 174‑182 in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)) uses `EvaluationStrategyFactory` to instantiate the appropriate strategy based on the `evaluation_method` key.
3. **Answer Parsing** – The evaluator calls `self.evaluation_strategy.extract_answer(content)`, passing the raw model output to the strategy.
4. **Validation** – The extracted string is compared against ground truth; `None` indicates a parsing failure.

## The Three Built-In Answer Extraction Strategies

All concrete strategies inherit from the abstract **`EvaluationStrategy`** class defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) (lines 14‑27). Each implements the `extract_answer(self, llm_output: str) -> Optional[str]` method to handle specific formatting conventions.

### PatternMatchingStrategy for Natural Language Responses

The **`PatternMatchingStrategy`** (lines 38‑104) applies a cascading series of regular expressions to locate answer tokens (typically A‑D) within free-text responses. This strategy handles both English and Chinese phrasing through default patterns such as:

- `r"correct answer is:\n\n([A-D])."`
- `r"答案是:\s?([A-D])"`
- `r"([A-D]). "`

The implementation iterates through the pattern list, returning the first captured group that matches:

```python
def extract_answer(self, llm_output: str) -> Optional[str]:
    if not self.validate_output(llm_output):
        return None
    for pattern in self.patterns:
        match = re.search(pattern, llm_output)
        if match:
            return match.group(1).strip()
    return None

```

### BoxExtractionStrategy for Mathematical Expressions

For models that output LaTeX-formatted answers, the **`BoxExtractionStrategy`** (lines 110‑136) detects `\box{A}` or `\boxed{A}` wrappers. This strategy uses targeted regex patterns to extract the contents of LaTeX box commands, making it ideal for mathematical reasoning benchmarks where answers are explicitly delimited.

### CustomRegexStrategy for User-Defined Rules

The **`CustomRegexStrategy`** (lines 138‑161) accepts arbitrary regex lists via configuration. Users supply patterns through the `strategy_config` field, enabling extraction logic for domain-specific formats without modifying source code.

## Code Implementation in the Evaluator

The actual invocation occurs in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) at lines 108‑114. After receiving the LLM completion, the evaluator delegates parsing to the strategy instance:

```python
content = message.content                # raw model output

predicted_answer = self.evaluation_strategy.extract_answer(content)

is_correct = (
    False if predicted_answer is None
    else predicted_answer.strip().upper() == correct_answer
)

```

This single call to `extract_answer` serves as the sole parsing gateway, ensuring consistent error handling across all strategy types.

## Configuring Answer Extraction Methods

Configuration resides in YAML or JSON files, where the `evaluation` section specifies the method and optional parameters. The `Config` class automatically builds the strategy via `EvaluationStrategyFactory` (lines 163‑191).

**Default pattern matching configuration:**

```yaml
evaluation:
  evaluation_method: pattern

```

**Custom regex configuration:**

```yaml
evaluation:
  evaluation_method: custom_regex
  strategy_config:
    patterns:
      - "final answer[:\\s]*([A-D])"
      - "答[案]為[:\\s]*([A-D])"

```

The factory registry maps strategy names to classes, allowing the `Evaluator` to remain agnostic of specific parsing implementations.

## Extending the Extraction System

Developers can introduce new extraction logic by subclassing `EvaluationStrategy` and implementing two required methods:

1. `extract_answer(self, llm_output: str) -> Optional[str]`
2. `get_strategy_name() -> str`

After implementation, register the class with `EvaluationStrategyFactory.register_strategy` to make it available via configuration:

```python
from twinkle_eval.evaluation_strategies import EvaluationStrategy, EvaluationStrategyFactory

class MyStrategy(EvaluationStrategy):
    def extract_answer(self, llm_output: str) -> Optional[str]:
        # Custom parsing logic

        return parsed
    
    def get_strategy_name(self) -> str:
        return "my_strategy"

EvaluationStrategyFactory.register_strategy(MyStrategy)

```

## Summary

- Twinkle Eval isolates answer extraction through a **strategy pattern** implemented in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py).
- Three built-in strategies handle **regex pattern matching**, **LaTeX box detection**, and **custom user-defined regex**.
- The **`Evaluator`** class delegates all parsing to `self.evaluation_strategy.extract_answer()`, centralizing error handling.
- Configuration via **`evaluation_method`** and **`strategy_config`** enables zero-code switching between extraction approaches.
- The factory architecture supports **extensible registration** of new strategies for domain-specific formats.

## Frequently Asked Questions

### What extraction methods does Twinkle Eval support by default?

Twinkle Eval provides three built-in methods: `PatternMatchingStrategy` for natural language multiple-choice answers, `BoxExtractionStrategy` for LaTeX boxed expressions, and `CustomRegexStrategy` for user-supplied regular expressions. All are defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py).

### How do I configure a custom regex pattern for answer extraction?

Set `evaluation_method: custom_regex` in your configuration file and provide a `strategy_config` block containing a `patterns` list. The `EvaluationStrategyFactory` instantiates `CustomRegexStrategy` with your specified regex, which then applies them in order until a match is found.

### Where does the answer extraction logic execute in the evaluation pipeline?

The extraction occurs in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) at lines 108‑114, where the `Evaluator` calls `self.evaluation_strategy.extract_answer(content)` immediately after receiving the LLM response. This single integration point ensures all parsing happens through the configured strategy.

### Can I implement a custom extraction strategy for specialized formats?

Yes. Subclass `EvaluationStrategy`, implement `extract_answer` and `get_strategy_name`, then register your class using `EvaluationStrategyFactory.register_strategy`. Once registered, reference your strategy by name in the `evaluation_method` configuration field.