How to Configure System Prompts for Different Languages in Twinkle Eval

To configure system prompts for different languages in Twinkle Eval, define language-specific prompts under evaluation.system_prompt in your configuration file and map dataset paths to specific languages using evaluation.datasets_prompt_map, falling back to Chinese (zh) when no mapping exists.

Twinkle Eval, the open-source LLM evaluation framework from the ai-twinkle/eval repository, enables multilingual benchmarking by allowing per-language system prompts. This flexibility ensures that your evaluation instructions match the language of the dataset, improving model performance assessment across Chinese, English, French, and other custom languages.

Configuration Keys for Multilingual Support

Twinkle Eval uses two primary configuration keys to manage language-specific system prompts. These are defined in your config.yaml and validated at runtime.

evaluation.system_prompt

The evaluation.system_prompt key holds the actual instruction text for each language. When the evaluation method is set to "box" (the default for multiple-choice tasks), this prompt is inserted as a system message before the user question. In twinkle_eval/models.py, the _build_messages() method extracts the appropriate prompt based on the language key:

sys_prompt_cfg = eval_config.get("system_prompt", {})
if isinstance(sys_prompt_cfg, dict):
    sys_prompt = sys_prompt_cfg.get(prompt_lang, sys_prompt_cfg.get("zh", ""))
else:
    sys_prompt = sys_prompt_cfg

evaluation.datasets_prompt_map

The evaluation.datasets_prompt_map key maps dataset directory paths to language codes. This tells Twinkle Eval which system prompt to use for each dataset. If a dataset path is not listed, the framework defaults to "zh" (Chinese). The resolution occurs in twinkle_eval/main.py inside the _evaluate_dataset() method:

prompt_map = self.config["evaluation"].get("datasets_prompt_map", {})
dataset_lang = prompt_map.get(dataset_path, "zh")

How Language Selection Works

The multilingual workflow follows a three-stage pipeline from configuration to LLM invocation.

Configuration Validation

When Twinkle Eval starts, the ConfigurationManager loads your YAML file and validates the structure. The datasets_prompt_map is checked in twinkle_eval/validators.py to ensure it is a dictionary of string-to-string pairs, preventing runtime errors from malformed mappings.

Dataset Language Resolution

For each dataset being evaluated, the Main._evaluate_dataset() function looks up the dataset's path in datasets_prompt_map. The resulting language code (e.g., "en", "fr", "zh") is passed down the call chain to the model interface.

Message Construction

Inside twinkle_eval/models.py, the OpenAIModel._build_messages() function receives the language code and retrieves the corresponding prompt from the system_prompt dictionary. This system message is prepended to the conversation history only when using the "box" evaluation method. Finally, OpenAIModel.call() transmits the complete message list to your OpenAI-compatible endpoint.

Practical Configuration Examples

Setting Up French and English Prompts

Add new languages by extending the system_prompt dictionary and mapping datasets accordingly:

# config.yaml

evaluation:
  evaluation_method: "box"
  system_prompt:
    zh: |
      使用者將提供一個題目，並附上選項 A、B、C、D
      請選出最符合的選項，輸出格式 \box{選項}
    en: |
      The user will provide a question with options A‑D.
      Choose the best option and output it as \box{Option}
    fr: |
      L'utilisateur fournit une question avec les options A‑D.
      Sélectionnez la meilleure option et renvoyez‑la sous la forme \box{Option}
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/french_mcq/": "fr"

When processing files under datasets/french_mcq/, Twinkle Eval automatically sends the French system prompt to the LLM.

Command-Line Language Overrides

For ad-hoc testing without modifying the config file, use the --benchmark-prompt flag to force a specific language:

twinkle-eval --config config.yaml --benchmark-prompt "fr"

This overrides the datasets_prompt_map lookup and passes "fr" directly as prompt_lang to the model.

Minimal Inline Configuration

For quick experiments, a minimal configuration requires only the prompt dictionary and a single mapping:

evaluation:
  system_prompt:
    zh: "中文提示 ..."
    en: "English prompt ..."
  datasets_prompt_map:
    "datasets/example/": "en"

Core Implementation Files

The multilingual system prompt logic spans four key files in the ai-twinkle/eval repository:

• twinkle_eval/config.py – Loads the YAML configuration, applies defaults, and instantiates LLM strategies.
• twinkle_eval/validators.py – Validates that datasets_prompt_map conforms to the required dictionary schema.
• twinkle_eval/main.py – Contains _evaluate_dataset(), which resolves the language for each dataset path.
• twinkle_eval/models.py – Houses _build_messages(), which selects and injects the language-specific system prompt.

Summary

• Define prompts using the evaluation.system_prompt dictionary, with language codes as keys (e.g., zh, en, fr).
• Map datasets to languages via evaluation.datasets_prompt_map, using the dataset directory path as the key.
• Default behavior falls back to Chinese (zh) when a dataset is not explicitly mapped.
• Method restriction means system prompts are only injected when evaluation_method is set to "box".
• Override capability exists via the --benchmark-prompt command-line flag for testing.

Frequently Asked Questions

What happens if a dataset is not listed in datasets_prompt_map?

Twinkle Eval defaults to the "zh" language key. If you have not defined a zh prompt in system_prompt, it falls back to an empty string. This logic is handled in twinkle_eval/main.py during the dataset evaluation loop.

Can I use the same system prompt for all languages?

Yes. If system_prompt is defined as a string rather than a dictionary, Twinkle Eval uses that single string for all evaluations regardless of language. However, using a dictionary with explicit language keys is recommended for multilingual accuracy.

Does this configuration work with evaluation methods other than box?

No. The system prompt injection logic in twinkle_eval/models.py specifically checks for the "box" evaluation method. Other methods may handle prompts differently or ignore the system prompt configuration entirely.

How do I add support for a language not currently in the template?

Simply add a new key-value pair to the system_prompt dictionary in your config (e.g., ja: for Japanese), then map your Japanese datasets to "ja" in datasets_prompt_map. Twinkle Eval accepts any string as a language key, so you are not limited to ISO codes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}