# PatternMatchingStrategy vs BoxExtractionStrategy: Understanding the Difference Between Pattern and Box Evaluation Strategies in Twinkle Eval

> Understand the difference between PatternMatchingStrategy and BoxExtractionStrategy in Twinkle Eval. Learn how each strategy finds answers within LLM outputs.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: deep-dive
- Published: 2026-02-23

---

**PatternMatchingStrategy scans LLM outputs using 40+ flexible regex patterns to find answers anywhere in the text, while BoxExtractionStrategy only extracts answers explicitly wrapped in LaTeX box commands (`\box{}` or `\boxed{}`).**

The `ai-twinkle/eval` repository provides a modular evaluation framework for benchmarking language models, where the **difference between pattern and box evaluation strategies** determines how raw model outputs are parsed into extractable answers. These two built-in strategies in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) serve distinct purposes: one handles unstructured natural language responses, while the other enforces strict formatting for mathematical and code evaluations.

## Core Differences Between Pattern and Box Evaluation Strategies

The fundamental distinction lies in **permissiveness versus structure**. PatternMatchingStrategy employs a broad, language-agnostic approach capable of recognizing answers phrased in English, Chinese, or custom formats. BoxExtractionStrategy operates as a narrow, format-driven extractor that requires models to delimit answers using specific LaTeX syntax.

- **PatternMatchingStrategy**: Uses 40+ default regex patterns covering variations like *"correct answer is: B"*, *"答案是：C"*, and *"答案 (D) 正確"* (lines 45-84 in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py)).
- **BoxExtractionStrategy**: Relies on two strict patterns matching `\box{X}` or `\boxed{X}` where `X` is A-D (lines 13-14 in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py)).

Both strategies implement the `extract_answer` method, but PatternMatchingStrategy iterates through extensive linguistic variations while BoxExtractionStrategy validates precise LaTeX formatting.

## How PatternMatchingStrategy Works

PatternMatchingStrategy functions as a **broad-spectrum answer extractor** designed for general-purpose benchmarks where models may respond in plain prose or informal wording. The strategy's `extract_answer` method iterates over `self.patterns`, executing `re.search` on the entire output and returning the first captured group (`match.group(1)`).

The default implementation includes regexes that handle:
- Colon-separated answers (*"The answer is: A"*)
- Parenthetical formats (*"Answer (B)"*)
- Multilingual phrasing (*"正确答案为 C"*)

```python
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

# Factory instantiates PatternMatchingStrategy (see lines 66-68)

pattern_strategy = EvaluationStrategyFactory.create_strategy("pattern")

llm_output = """
After analyzing the options, I believe the correct answer is:

B
"""

answer = pattern_strategy.extract_answer(llm_output)
print(answer)  # → "B"

```

## How BoxExtractionStrategy Works

BoxExtractionStrategy serves as a **structured format validator** for evaluations requiring explicit answer demarcation. This strategy only succeeds when the model wraps its final answer in LaTeX box commands, making it ideal for math-oriented prompts where unambiguous extraction is critical.

The strategy uses minimal regex patterns to match:
- `\box{X}` 
- `\boxed{X}`

```python
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

box_strategy = EvaluationStrategyFactory.create_strategy("box")

llm_output = r"""
We compute the integral and place the final result in a box:

\boxed{C}
"""

answer = box_strategy.extract_answer(llm_output)
print(answer)  # → "C"

```

When the model outputs plain text without LaTeX delimiters, `extract_answer` returns `None`, effectively rejecting improperly formatted responses.

## When to Use Each Strategy

**Choose PatternMatchingStrategy** when evaluating:
- General knowledge benchmarks with free-form responses
- Multilingual datasets requiring flexible parsing
- Legacy datasets with inconsistent answer formatting

**Choose BoxExtractionStrategy** when evaluating:
- Mathematical reasoning tasks requiring `\boxed{}` notation
- Code generation benchmarks with strict output constraints
- Prompts explicitly instructing models to use LaTeX boxes

## Customizing Evaluation Strategies

Both strategies expose the `add_pattern` method for extending default behavior, though their design intents differ. PatternMatchingStrategy welcomes permissive additions to handle new linguistic variations, while BoxExtractionStrategy extensions typically add alternative delimiters.

**Adding custom regex to PatternMatchingStrategy:**

```python
pattern_strategy = EvaluationStrategyFactory.create_strategy("pattern")
pattern_strategy.add_pattern(r"答案是答案：\s*([A-D])")

# Now handles additional Chinese phrasing variations

```

**Adding custom delimiters to BoxExtractionStrategy:**

```python
box_strategy = EvaluationStrategyFactory.create_strategy("box")
box_strategy.add_pattern(r"\\begin{answer}([A-D])\\end{answer}")

# Enables extraction from custom LaTeX environments

```

## Summary

- **PatternMatchingStrategy** extracts answers from anywhere in the text using 40+ regex patterns supporting multiple languages and phrasings, defined in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) lines 45-84.
- **BoxExtractionStrategy** only extracts answers wrapped in `\box{}` or `\boxed{}` LaTeX commands, using the strict patterns defined in lines 13-14.
- Both strategies are instantiated via `EvaluationStrategyFactory.create_strategy()` and implement the `extract_answer` method.
- Use PatternMatchingStrategy for general benchmarks and BoxExtractionStrategy for math/code evaluations requiring structured output.
- Extend either strategy using the `add_pattern` method to handle custom formats.

## Frequently Asked Questions

### Can I use both PatternMatchingStrategy and BoxExtractionStrategy in the same evaluation?

Yes. You can instantiate both strategies separately using `EvaluationStrategyFactory.create_strategy()` and apply them sequentially or conditionally based on the prompt type. For mixed datasets, consider using PatternMatchingStrategy as a fallback when BoxExtractionStrategy returns `None`.

### How do I add support for new languages to PatternMatchingStrategy?

Call the `add_pattern` method with a regex that captures the target language's answer phrasing. For example, to support Spanish responses like *"La respuesta correcta es: A"*, add the pattern `r"La respuesta correcta es:\s*([A-D])"` to your strategy instance.

### Why does BoxExtractionStrategy fail to extract answers from plain text responses?

BoxExtractionStrategy is intentionally restrictive. According to the source code in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) lines 13-14, it only matches the specific LaTeX box syntax `\box{X}` or `\boxed{X}`. If the model outputs "The answer is B" without LaTeX delimiters, the strategy correctly returns `None` to enforce formatting compliance.

### Which strategy should I choose for evaluating mathematical word problems?

Use **BoxExtractionStrategy** for math problems. By prompting the model to place final answers inside `\boxed{}` commands, you eliminate ambiguity from explanatory text and ensure precise extraction. This approach aligns with standard mathematical typesetting practices and reduces false positives from intermediate calculations.