# How Option Randomization in Twinkle Eval Reduces Model Bias During Evaluation

> Learn how Twinkle Eval uses option randomization to reduce LLM bias by shuffling answers. Force models to evaluate content not position for fairer AI evaluation.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: deep-dive
- Published: 2026-02-23

---

**Twinkle Eval mitigates positional bias by shuffling multiple-choice options before sending them to the LLM, forcing the model to evaluate content rather than rely on fixed answer positions.**

Option randomization in Twinkle Eval is a critical feature for unbiased LLM benchmarking in the `ai-twinkle/eval` repository. When evaluating multiple-choice questions, large language models often exhibit positional bias—tendencies to favor specific answer positions like "A" or the first option. The repository implements a configurable shuffling mechanism that randomizes option orderings to eliminate this bias and produce more accurate performance metrics.

## Understanding Positional Bias in LLM Evaluation

Positional bias occurs when LLMs consistently select answers based on their position in the prompt rather than their semantic content. Research shows that models often favor the first or last option in multiple-choice formats, or develop spurious correlations during training. Without intervention, evaluation results reflect these artifacts rather than true reasoning capability, leading to inflated or deflated accuracy scores that misrepresent model performance.

## How Twinkle Eval Implements Option Randomization

### Configuration and Activation

The feature is controlled through the `shuffle_options` flag in the evaluation configuration. In [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) (lines 99-103), this parameter defaults to `False` within the evaluation section of [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml). Setting `shuffle_options: true` activates randomization for the entire evaluation run, applying the shuffle to every multiple-choice question processed.

### The Shuffling Algorithm

When enabled, the shuffling logic executes within the `Evaluator.evaluate_file` method in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py). At lines 75-78, the code checks the configuration flag and conditionally calls `shuffle_question_options` for each question before sending it to the LLM.

The `shuffle_question_options` function (lines 46-60) performs the following steps:

- Extracts the existing option texts from keys A through D into a Python list
- Records the text of the currently correct answer before shuffling
- Randomizes the list order using `random.shuffle(options)` at line 50
- Rebuilds the question dictionary by assigning shuffled texts back to keys A-D in the new order
- Updates the `"answer"` field to point to the new key (A, B, C, or D) that now contains the correct text (lines 54-60)

This approach ensures that while the option labels remain consistent (always A-D), the underlying content associated with each label changes randomly for every evaluation run.

## Practical Implementation: Enabling Option Randomization

To activate option randomization via configuration file:

```yaml

# config.yaml

evaluation:
  repeat_runs: 3                # Optional: run multiple times with different shuffles

  shuffle_options: true         # Enable option randomization

  evaluation_method: "exact_match"
  dataset_paths:
    - "datasets/zh_mcqa"

```

Execute the evaluation via the CLI entry point:

```bash
python -m twinkle_eval.main --config config.yaml

```

Running with `repeat_runs: 3` and `shuffle_options: true` generates three separate result files, each with different option orderings. Aggregating accuracies across these runs provides a bias-mitigated final score.

For programmatic control:

```python
from twinkle_eval.evaluators import Evaluator
from twinkle_eval.config import ConfigurationManager

# Load and override configuration

cfg = ConfigurationManager().load_config()
cfg["evaluation"]["shuffle_options"] = True  # Force enable at runtime

evaluator = Evaluator(
    llm=cfg["llm_instance"],
    evaluation_strategy=cfg["evaluation_strategy_instance"],
    config=cfg,
)

# Evaluate with shuffled options

file_path, accuracy, result_path = evaluator.evaluate_file(
    file_path="datasets/zh_mcqa/sample.jsonl",
    timestamp="20240223_001"
)
print(f"Accuracy with option randomization: {accuracy:.2%}")

```

## Summary

- **Positional bias** occurs when LLMs exploit fixed answer positions rather than content, distorting evaluation metrics.
- Twinkle Eval's `shuffle_options` feature, defined in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py), enables randomization of multiple-choice options.
- The `shuffle_question_options` function in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) (lines 46-60) implements the shuffle using `random.shuffle`, remapping correct answers to new positions while preserving label consistency.
- Enabling this feature forces models to evaluate semantic content rather than positional heuristics, producing more accurate capability assessments.
- Running multiple evaluation passes with different random seeds further smooths residual ordering effects.

## Frequently Asked Questions

### Does option randomization affect evaluation reproducibility?

While individual evaluation runs produce different option orderings and may yield slightly varying accuracy scores per run, the aggregate results across multiple runs become more stable and representative of true model capability. For debugging purposes, you can seed Python's random number generator before running Twinkle Eval to obtain deterministic shuffles, though the repository defaults to non-deterministic randomization for unbiased evaluation.

### What is the performance impact of enabling shuffle_options?

The performance overhead is negligible. The shuffling operation occurs once per question using Python's built-in `random.shuffle`, which executes in microseconds. Compared to the latency of LLM inference calls, the time required to randomize options in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) is effectively zero and does not impact overall evaluation throughput.

### Can option randomization be used with custom evaluation metrics?

Yes. The shuffling logic in `shuffle_question_options` modifies the input data before it reaches the evaluation strategy. Since the function updates both the option texts and the correct answer key to maintain consistency, any downstream evaluation method—including exact match, semantic similarity, or custom metrics—receives properly formatted data. The randomization is transparent to the evaluation logic itself.

### How does Twinkle Eval handle answer key mapping after shuffling?

The `shuffle_question_options` function in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) (lines 54-60) preserves answer integrity by first storing the text of the correct answer before shuffling. After randomizing the option list with `random.shuffle`, it iterates through the shuffled options to identify which new label (A, B, C, or D) now contains the correct text, then updates the `"answer"` field in the question dictionary to point to this new key.