# How to Configure System Prompts for Different Languages in Twinkle Eval

> Easily configure system prompts for multiple languages in Twinkle Eval. Define language-specific prompts and map dataset paths for seamless multilingual evaluation.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**To configure system prompts for different languages in Twinkle Eval, define language-specific prompts under `evaluation.system_prompt` in your configuration file and map dataset paths to specific languages using `evaluation.datasets_prompt_map`, falling back to Chinese (`zh`) when no mapping exists.**

Twinkle Eval, the open-source LLM evaluation framework from the ai-twinkle/eval repository, enables multilingual benchmarking by allowing per-language system prompts. This flexibility ensures that your evaluation instructions match the language of the dataset, improving model performance assessment across Chinese, English, French, and other custom languages.

## Configuration Keys for Multilingual Support

Twinkle Eval uses two primary configuration keys to manage language-specific system prompts. These are defined in your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) and validated at runtime.

### evaluation.system_prompt

The `evaluation.system_prompt` key holds the actual instruction text for each language. When the **evaluation method** is set to `"box"` (the default for multiple-choice tasks), this prompt is inserted as a system message before the user question. In [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py), the `_build_messages()` method extracts the appropriate prompt based on the language key:

```python
sys_prompt_cfg = eval_config.get("system_prompt", {})
if isinstance(sys_prompt_cfg, dict):
    sys_prompt = sys_prompt_cfg.get(prompt_lang, sys_prompt_cfg.get("zh", ""))
else:
    sys_prompt = sys_prompt_cfg

```

### evaluation.datasets_prompt_map

The `evaluation.datasets_prompt_map` key maps **dataset directory paths** to language codes. This tells Twinkle Eval which system prompt to use for each dataset. If a dataset path is not listed, the framework defaults to `"zh"` (Chinese). The resolution occurs in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) inside the `_evaluate_dataset()` method:

```python
prompt_map = self.config["evaluation"].get("datasets_prompt_map", {})
dataset_lang = prompt_map.get(dataset_path, "zh")

```

## How Language Selection Works

The multilingual workflow follows a three-stage pipeline from configuration to LLM invocation.

### Configuration Validation

When Twinkle Eval starts, the `ConfigurationManager` loads your YAML file and validates the structure. The `datasets_prompt_map` is checked in [`twinkle_eval/validators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/validators.py) to ensure it is a dictionary of string-to-string pairs, preventing runtime errors from malformed mappings.

### Dataset Language Resolution

For each dataset being evaluated, the `Main._evaluate_dataset()` function looks up the dataset's path in `datasets_prompt_map`. The resulting language code (e.g., `"en"`, `"fr"`, `"zh"`) is passed down the call chain to the model interface.

### Message Construction

Inside [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py), the `OpenAIModel._build_messages()` function receives the language code and retrieves the corresponding prompt from the `system_prompt` dictionary. This system message is prepended to the conversation history only when using the `"box"` evaluation method. Finally, `OpenAIModel.call()` transmits the complete message list to your OpenAI-compatible endpoint.

## Practical Configuration Examples

### Setting Up French and English Prompts

Add new languages by extending the `system_prompt` dictionary and mapping datasets accordingly:

```yaml

# config.yaml

evaluation:
  evaluation_method: "box"
  system_prompt:
    zh: |
      使用者將提供一個題目，並附上選項 A、B、C、D
      請選出最符合的選項，輸出格式 \box{選項}
    en: |
      The user will provide a question with options A‑D.
      Choose the best option and output it as \box{Option}
    fr: |
      L'utilisateur fournit une question avec les options A‑D.
      Sélectionnez la meilleure option et renvoyez‑la sous la forme \box{Option}
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/french_mcq/": "fr"

```

When processing files under `datasets/french_mcq/`, Twinkle Eval automatically sends the French system prompt to the LLM.

### Command-Line Language Overrides

For ad-hoc testing without modifying the config file, use the `--benchmark-prompt` flag to force a specific language:

```bash
twinkle-eval --config config.yaml --benchmark-prompt "fr"

```

This overrides the `datasets_prompt_map` lookup and passes `"fr"` directly as `prompt_lang` to the model.

### Minimal Inline Configuration

For quick experiments, a minimal configuration requires only the prompt dictionary and a single mapping:

```yaml
evaluation:
  system_prompt:
    zh: "中文提示 ..."
    en: "English prompt ..."
  datasets_prompt_map:
    "datasets/example/": "en"

```

## Core Implementation Files

The multilingual system prompt logic spans four key files in the ai-twinkle/eval repository:

- **[`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)** – Loads the YAML configuration, applies defaults, and instantiates LLM strategies.
- **[`twinkle_eval/validators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/validators.py)** – Validates that `datasets_prompt_map` conforms to the required dictionary schema.
- **[`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py)** – Contains `_evaluate_dataset()`, which resolves the language for each dataset path.
- **[`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py)** – Houses `_build_messages()`, which selects and injects the language-specific system prompt.

## Summary

- **Define prompts** using the `evaluation.system_prompt` dictionary, with language codes as keys (e.g., `zh`, `en`, `fr`).
- **Map datasets** to languages via `evaluation.datasets_prompt_map`, using the dataset directory path as the key.
- **Default behavior** falls back to Chinese (`zh`) when a dataset is not explicitly mapped.
- **Method restriction** means system prompts are only injected when `evaluation_method` is set to `"box"`.
- **Override capability** exists via the `--benchmark-prompt` command-line flag for testing.

## Frequently Asked Questions

### What happens if a dataset is not listed in datasets_prompt_map?

Twinkle Eval defaults to the `"zh"` language key. If you have not defined a `zh` prompt in `system_prompt`, it falls back to an empty string. This logic is handled in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) during the dataset evaluation loop.

### Can I use the same system prompt for all languages?

Yes. If `system_prompt` is defined as a string rather than a dictionary, Twinkle Eval uses that single string for all evaluations regardless of language. However, using a dictionary with explicit language keys is recommended for multilingual accuracy.

### Does this configuration work with evaluation methods other than box?

No. The system prompt injection logic in [`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py) specifically checks for the `"box"` evaluation method. Other methods may handle prompts differently or ignore the system prompt configuration entirely.

### How do I add support for a language not currently in the template?

Simply add a new key-value pair to the `system_prompt` dictionary in your config (e.g., `ja:` for Japanese), then map your Japanese datasets to `"ja"` in `datasets_prompt_map`. Twinkle Eval accepts any string as a language key, so you are not limited to ISO codes.