How Option Randomization in Twinkle Eval Reduces Model Bias During Evaluation

Twinkle Eval mitigates positional bias by shuffling multiple-choice options before sending them to the LLM, forcing the model to evaluate content rather than rely on fixed answer positions.

Option randomization in Twinkle Eval is a critical feature for unbiased LLM benchmarking in the ai-twinkle/eval repository. When evaluating multiple-choice questions, large language models often exhibit positional bias—tendencies to favor specific answer positions like "A" or the first option. The repository implements a configurable shuffling mechanism that randomizes option orderings to eliminate this bias and produce more accurate performance metrics.

Understanding Positional Bias in LLM Evaluation

Positional bias occurs when LLMs consistently select answers based on their position in the prompt rather than their semantic content. Research shows that models often favor the first or last option in multiple-choice formats, or develop spurious correlations during training. Without intervention, evaluation results reflect these artifacts rather than true reasoning capability, leading to inflated or deflated accuracy scores that misrepresent model performance.

How Twinkle Eval Implements Option Randomization

Configuration and Activation

The feature is controlled through the shuffle_options flag in the evaluation configuration. In twinkle_eval/config.py (lines 99-103), this parameter defaults to False within the evaluation section of config.yaml. Setting shuffle_options: true activates randomization for the entire evaluation run, applying the shuffle to every multiple-choice question processed.

The Shuffling Algorithm

When enabled, the shuffling logic executes within the Evaluator.evaluate_file method in twinkle_eval/evaluators.py. At lines 75-78, the code checks the configuration flag and conditionally calls shuffle_question_options for each question before sending it to the LLM.

The shuffle_question_options function (lines 46-60) performs the following steps:

• Extracts the existing option texts from keys A through D into a Python list
• Records the text of the currently correct answer before shuffling
• Randomizes the list order using random.shuffle(options) at line 50
• Rebuilds the question dictionary by assigning shuffled texts back to keys A-D in the new order
• Updates the "answer" field to point to the new key (A, B, C, or D) that now contains the correct text (lines 54-60)

This approach ensures that while the option labels remain consistent (always A-D), the underlying content associated with each label changes randomly for every evaluation run.

Practical Implementation: Enabling Option Randomization

To activate option randomization via configuration file:

# config.yaml

evaluation:
  repeat_runs: 3                # Optional: run multiple times with different shuffles

  shuffle_options: true         # Enable option randomization

  evaluation_method: "exact_match"
  dataset_paths:
    - "datasets/zh_mcqa"

Execute the evaluation via the CLI entry point:

python -m twinkle_eval.main --config config.yaml

Running with repeat_runs: 3 and shuffle_options: true generates three separate result files, each with different option orderings. Aggregating accuracies across these runs provides a bias-mitigated final score.

For programmatic control:

from twinkle_eval.evaluators import Evaluator
from twinkle_eval.config import ConfigurationManager

# Load and override configuration

cfg = ConfigurationManager().load_config()
cfg["evaluation"]["shuffle_options"] = True  # Force enable at runtime

evaluator = Evaluator(
    llm=cfg["llm_instance"],
    evaluation_strategy=cfg["evaluation_strategy_instance"],
    config=cfg,
)

# Evaluate with shuffled options

file_path, accuracy, result_path = evaluator.evaluate_file(
    file_path="datasets/zh_mcqa/sample.jsonl",
    timestamp="20240223_001"
)
print(f"Accuracy with option randomization: {accuracy:.2%}")

Summary

• Positional bias occurs when LLMs exploit fixed answer positions rather than content, distorting evaluation metrics.
• Twinkle Eval's shuffle_options feature, defined in twinkle_eval/config.py, enables randomization of multiple-choice options.
• The shuffle_question_options function in twinkle_eval/evaluators.py (lines 46-60) implements the shuffle using random.shuffle, remapping correct answers to new positions while preserving label consistency.
• Enabling this feature forces models to evaluate semantic content rather than positional heuristics, producing more accurate capability assessments.
• Running multiple evaluation passes with different random seeds further smooths residual ordering effects.

Frequently Asked Questions

Does option randomization affect evaluation reproducibility?

While individual evaluation runs produce different option orderings and may yield slightly varying accuracy scores per run, the aggregate results across multiple runs become more stable and representative of true model capability. For debugging purposes, you can seed Python's random number generator before running Twinkle Eval to obtain deterministic shuffles, though the repository defaults to non-deterministic randomization for unbiased evaluation.

What is the performance impact of enabling shuffle_options?

The performance overhead is negligible. The shuffling operation occurs once per question using Python's built-in random.shuffle, which executes in microseconds. Compared to the latency of LLM inference calls, the time required to randomize options in twinkle_eval/evaluators.py is effectively zero and does not impact overall evaluation throughput.

Can option randomization be used with custom evaluation metrics?

Yes. The shuffling logic in shuffle_question_options modifies the input data before it reaches the evaluation strategy. Since the function updates both the option texts and the correct answer key to maintain consistency, any downstream evaluation method—including exact match, semantic similarity, or custom metrics—receives properly formatted data. The randomization is transparent to the evaluation logic itself.

How does Twinkle Eval handle answer key mapping after shuffling?

The shuffle_question_options function in twinkle_eval/evaluators.py (lines 54-60) preserves answer integrity by first storing the text of the correct answer before shuffling. After randomizing the option list with random.shuffle, it iterates through the shuffled options to identify which new label (A, B, C, or D) now contains the correct text, then updates the "answer" field in the question dictionary to point to this new key.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}