# Twinkle Eval Default Config Template Structure and Customization Guide

> Explore the Twinkle Eval default config template structure and learn how to customize it. Understand LLM settings, model parameters, and evaluation workflows with ease. Get started today!

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**The Twinkle Eval framework uses a YAML-driven configuration system where [`twinkle_eval/config.template.yaml`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.template.yaml) provides a complete skeleton of LLM settings, model parameters, and evaluation workflows, automatically applying sensible defaults via `ConfigurationManager._apply_defaults()` for any keys you omit.**

The **ai-twinkle/eval** repository delivers a flexible evaluation framework for Large Language Models that is entirely driven by configuration files. Understanding the **default config template structure** allows you to quickly adapt the system for diverse evaluation scenarios—from single-dataset debugging to multilingual batch processing with cloud-based result export. The configuration layer, implemented primarily in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py), validates your inputs and merges them with built-in defaults to ensure reliable execution.

## Default Config Template Layout

The reference template shipped at [`twinkle_eval/config.template.yaml`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.template.yaml) organizes settings into six logical sections. Each section controls a distinct aspect of the evaluation pipeline:

- **`llm_api`** – Connection parameters for the inference endpoint, including `base_url`, `api_key`, `type` (e.g., `openai` or `anthropic`), `api_rate_limit`, `max_retries`, and `timeout`.
- **`model`** – Generation hyperparameters such as `name`, `temperature`, `top_p`, `max_tokens`, `frequency_penalty`, `presence_penalty`, and `extra_body`.
- **`evaluation`** – Core workflow directives: `dataset_paths`, `evaluation_method` (`pattern` or `box`), multilingual `system_prompt` mappings, `datasets_prompt_map`, `repeat_runs`, `shuffle_options`, and `strategy_config`.
- **`environment`** – Informational metadata about hardware (`gpu_info`, `parallel_config`) and `system_info`; not used for execution logic but logged for reproducibility.
- **`logging`** – Console verbosity controlled via the `level` key.
- **`google_services`** (optional) – Credentials and toggles for exporting results to Google Drive or Google Sheets, including `auth_method`, `credentials_file`, and destination IDs.

## How the Configuration System Processes Your File

When you launch an evaluation with `python -m twinkle_eval.main --config my_config.yaml`, the `ConfigurationManager` class orchestrates a four-stage pipeline:

1. **Load and syntax validation** – `ConfigurationManager.load_config()` reads the file and validates YAML syntax through `validate_yaml_syntax`, then checks structural integrity via `validate_config_structure`.
2. **Apply defaults** – The private method `_apply_defaults()` (lines 65-108 in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)) injects fallback values for any omitted optional keys, such as `type: openai` for the LLM API, `repeat_runs: 1`, or `"Unknown"` for GPU model names.
3. **Dataset verification** – The `DatasetValidator` class confirms that every path in `evaluation.dataset_paths` exists and contains valid files.
4. **Component instantiation** – Factory methods `LLMFactory.create_llm` and `EvaluationStrategyFactory.create_strategy` build the runtime objects using the resolved configuration.

This design ensures that even a minimal user-supplied file expands into a fully functional configuration at runtime.

## Customizing the Config for Different Evaluation Scenarios

You can tailor the framework to specific needs by overriding targeted sections while relying on defaults for the rest.

### Switching LLM Providers or Hosts

Modify the `llm_api` section to point to different endpoints or authentication schemes:

```yaml
llm_api:
  base_url: "https://my-vllm.server/v1"
  api_key: "my-secret-key"
  type: "openai"
  disable_ssl_verify: false
  max_retries: 3

```

### Adjusting Model Hyperparameters

Control generation behavior through the `model` block:

```yaml
model:
  name: "gpt-4o-mini"
  temperature: 0.2
  top_p: 0.9
  max_tokens: 8192
  frequency_penalty: 0.0

```

### Changing Evaluation Methods

Set `evaluation.evaluation_method` to switch between parsing strategies:

- **`box`** – Expects answers wrapped in LaTeX-style `\box{}` format.
- **`pattern`** – Uses regex-based extraction defined in the strategy implementation.

```yaml
evaluation:
  evaluation_method: "pattern"

```

### Configuring Multilingual System Prompts

Define language-specific prompts under `system_prompt` and map datasets to languages via `datasets_prompt_map`:

```yaml
evaluation:
  system_prompt:
    en: |
      The user will provide a multiple-choice question.
      Output only the letter corresponding to the correct answer.
    zh: |
      用户将提供一个多选题。请输出正确选项的字母。
    fr: |
      L'utilisateur fournira une question à choix multiples.
      Répondez uniquement avec la lettre de la bonne réponse.
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/chinese/": "zh"
    "datasets/french/": "fr"

```

### Managing Datasets and Repeat Runs

Add paths to `dataset_paths` for new corpora, and adjust robustness settings:

```yaml
evaluation:
  dataset_paths:
    - "datasets/hellaswag/"
    - "datasets/mmlu/"
  repeat_runs: 5
  shuffle_options: true

```

Setting `repeat_runs` above 1 executes the evaluation multiple times per dataset, while `shuffle_options: true` randomizes answer ordering to detect position bias.

### Injecting Strategy-Specific Configurations

Some evaluation strategies accept custom arguments through `strategy_config`, which are passed unchanged to the strategy constructor:

```yaml
evaluation:
  strategy_config:
    temperature_schedule: [0.0, 0.2, 0.5]
    custom_regex_flags: "IGNORECASE"

```

### Enabling Google Services Integration

Export logs and results automatically by configuring the optional `google_services` block:

```yaml
google_services:
  google_drive:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    log_folder_id: "1a2b3c..."
  google_sheets:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    spreadsheet_id: "1XyZ..."
    sheet_name: "EvalResults"

```

When enabled, [`twinkle_eval/google_services.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/google_services.py) uploads execution logs and result rows upon completion.

## Complete Configuration Examples

### Minimal Single-Dataset Setup

This example runs the MMLU dataset three times using the `box` evaluation method against a local vLLM server:

```yaml
llm_api:
  base_url: "http://localhost:8000/v1"
  api_key: "dummy"
  type: "openai"

model:
  name: "gpt-4o-mini"
  temperature: 0.0
  max_tokens: 2048

evaluation:
  dataset_paths:
    - "datasets/mmlu/"
  evaluation_method: "box"
  system_prompt:
    en: |
      The user will provide a multiple-choice question.
      Output the answer as \box{Option}.
  repeat_runs: 3
  shuffle_options: false

logging:
  level: "INFO"

```

Execute with:

```bash
python -m twinkle_eval.main --config my_config.yaml

```

### Multilingual Batch Evaluation with Cloud Export

This configuration evaluates English and Chinese datasets with language-specific prompts and streams results to Google Sheets:

```yaml
llm_api:
  base_url: "https://api.openai.com/v1"
  api_key: "${OPENAI_API_KEY}"
  type: "openai"

model:
  name: "gpt-4"
  temperature: 0.1

evaluation:
  dataset_paths:
    - "datasets/mmlu/"
    - "datasets/chinese/"
  evaluation_method: "pattern"
  system_prompt:
    en: "Answer with the correct option letter only."
    zh: "仅输出正确选项的字母。"
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/chinese/": "zh"
  repeat_runs: 1
  shuffle_options: true

google_services:
  google_sheets:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    spreadsheet_id: "1A2B3C4D5E"
    sheet_name: "MultilingualEval"

logging:
  level: "DEBUG"

```

## Summary

- The **default config template** at [`twinkle_eval/config.template.yaml`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.template.yaml) defines six primary sections: `llm_api`, `model`, `evaluation`, `environment`, `logging`, and `google_services`.
- The **runtime configuration** is built by `ConfigurationManager.load_config()`, which validates syntax, applies defaults via `_apply_defaults()` (lines 65-108), and verifies dataset paths through `DatasetValidator`.
- **Customization** focuses on the `evaluation` section for datasets, prompts, and methods, while `llm_api` and `model` control inference behavior.
- **Google integrations** are optional and configured under `google_services` for automatic export to Drive or Sheets.
- **Factory classes** (`LLMFactory`, `EvaluationStrategyFactory`) instantiate components based on the resolved configuration, supporting methods like `box` and `pattern`.

## Frequently Asked Questions

### What happens if I omit optional keys in my config file?

The `ConfigurationManager` automatically injects sensible defaults through the `_apply_defaults()` method in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py). For example, if you omit `llm_api.type`, it defaults to `"openai"`; if you omit `repeat_runs`, it defaults to `1`. The effective configuration always contains every key required for execution.

### How do I validate my configuration before running an evaluation?

The framework validates your file automatically during the loading phase. `validate_yaml_syntax` checks for parse errors, while `validate_config_structure` ensures required sections exist. Additionally, `DatasetValidator` verifies that all paths in `evaluation.dataset_paths` point to existing directories with valid files. Run the CLI with `--config` to trigger these checks immediately.

### Can I use environment variables instead of hardcoding API keys?

While the raw template shows hardcoded values for clarity, you can use shell environment variable substitution before passing the file to the framework, or modify the `ConfigurationManager` in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) to resolve `${VAR}` syntax. The `google_services` sections explicitly expect file paths for credentials (e.g., `credentials_file: "service_account.json"`), keeping secrets out of the main config.

### What evaluation methods are supported and how do I switch between them?

The framework supports two primary methods: **`box`** (LaTeX-style `\box{}` wrapping) and **`pattern`** (regex-based extraction). Set `evaluation.evaluation_method` to either value. The `EvaluationStrategyFactory.create_strategy` method in [`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py) instantiates the appropriate parser based on this key.