Twinkle Eval Default Config Template Structure and Customization Guide

The Twinkle Eval framework uses a YAML-driven configuration system where twinkle_eval/config.template.yaml provides a complete skeleton of LLM settings, model parameters, and evaluation workflows, automatically applying sensible defaults via ConfigurationManager._apply_defaults() for any keys you omit.

The ai-twinkle/eval repository delivers a flexible evaluation framework for Large Language Models that is entirely driven by configuration files. Understanding the default config template structure allows you to quickly adapt the system for diverse evaluation scenarios—from single-dataset debugging to multilingual batch processing with cloud-based result export. The configuration layer, implemented primarily in twinkle_eval/config.py, validates your inputs and merges them with built-in defaults to ensure reliable execution.

Default Config Template Layout

The reference template shipped at twinkle_eval/config.template.yaml organizes settings into six logical sections. Each section controls a distinct aspect of the evaluation pipeline:

• llm_api – Connection parameters for the inference endpoint, including base_url, api_key, type (e.g., openai or anthropic), api_rate_limit, max_retries, and timeout.
• model – Generation hyperparameters such as name, temperature, top_p, max_tokens, frequency_penalty, presence_penalty, and extra_body.
• evaluation – Core workflow directives: dataset_paths, evaluation_method (pattern or box), multilingual system_prompt mappings, datasets_prompt_map, repeat_runs, shuffle_options, and strategy_config.
• environment – Informational metadata about hardware (gpu_info, parallel_config) and system_info; not used for execution logic but logged for reproducibility.
• logging – Console verbosity controlled via the level key.
• google_services (optional) – Credentials and toggles for exporting results to Google Drive or Google Sheets, including auth_method, credentials_file, and destination IDs.

How the Configuration System Processes Your File

When you launch an evaluation with python -m twinkle_eval.main --config my_config.yaml, the ConfigurationManager class orchestrates a four-stage pipeline:

1. Load and syntax validation – ConfigurationManager.load_config() reads the file and validates YAML syntax through validate_yaml_syntax, then checks structural integrity via validate_config_structure.
2. Apply defaults – The private method _apply_defaults() (lines 65-108 in twinkle_eval/config.py) injects fallback values for any omitted optional keys, such as type: openai for the LLM API, repeat_runs: 1, or "Unknown" for GPU model names.
3. Dataset verification – The DatasetValidator class confirms that every path in evaluation.dataset_paths exists and contains valid files.
4. Component instantiation – Factory methods LLMFactory.create_llm and EvaluationStrategyFactory.create_strategy build the runtime objects using the resolved configuration.

This design ensures that even a minimal user-supplied file expands into a fully functional configuration at runtime.

Customizing the Config for Different Evaluation Scenarios

You can tailor the framework to specific needs by overriding targeted sections while relying on defaults for the rest.

Switching LLM Providers or Hosts

Modify the llm_api section to point to different endpoints or authentication schemes:

llm_api:
  base_url: "https://my-vllm.server/v1"
  api_key: "my-secret-key"
  type: "openai"
  disable_ssl_verify: false
  max_retries: 3

Adjusting Model Hyperparameters

Control generation behavior through the model block:

model:
  name: "gpt-4o-mini"
  temperature: 0.2
  top_p: 0.9
  max_tokens: 8192
  frequency_penalty: 0.0

Changing Evaluation Methods

Set evaluation.evaluation_method to switch between parsing strategies:

• box – Expects answers wrapped in LaTeX-style \box{} format.
• pattern – Uses regex-based extraction defined in the strategy implementation.

evaluation:
  evaluation_method: "pattern"

Configuring Multilingual System Prompts

Define language-specific prompts under system_prompt and map datasets to languages via datasets_prompt_map:

evaluation:
  system_prompt:
    en: |
      The user will provide a multiple-choice question.
      Output only the letter corresponding to the correct answer.
    zh: |
      用户将提供一个多选题。请输出正确选项的字母。
    fr: |
      L'utilisateur fournira une question à choix multiples.
      Répondez uniquement avec la lettre de la bonne réponse.
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/chinese/": "zh"
    "datasets/french/": "fr"

Managing Datasets and Repeat Runs

Add paths to dataset_paths for new corpora, and adjust robustness settings:

evaluation:
  dataset_paths:
    - "datasets/hellaswag/"
    - "datasets/mmlu/"
  repeat_runs: 5
  shuffle_options: true

Setting repeat_runs above 1 executes the evaluation multiple times per dataset, while shuffle_options: true randomizes answer ordering to detect position bias.

Injecting Strategy-Specific Configurations

Some evaluation strategies accept custom arguments through strategy_config, which are passed unchanged to the strategy constructor:

evaluation:
  strategy_config:
    temperature_schedule: [0.0, 0.2, 0.5]
    custom_regex_flags: "IGNORECASE"

Enabling Google Services Integration

Export logs and results automatically by configuring the optional google_services block:

google_services:
  google_drive:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    log_folder_id: "1a2b3c..."
  google_sheets:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    spreadsheet_id: "1XyZ..."
    sheet_name: "EvalResults"

When enabled, twinkle_eval/google_services.py uploads execution logs and result rows upon completion.

Complete Configuration Examples

Minimal Single-Dataset Setup

This example runs the MMLU dataset three times using the box evaluation method against a local vLLM server:

llm_api:
  base_url: "http://localhost:8000/v1"
  api_key: "dummy"
  type: "openai"

model:
  name: "gpt-4o-mini"
  temperature: 0.0
  max_tokens: 2048

evaluation:
  dataset_paths:
    - "datasets/mmlu/"
  evaluation_method: "box"
  system_prompt:
    en: |
      The user will provide a multiple-choice question.
      Output the answer as \box{Option}.
  repeat_runs: 3
  shuffle_options: false

logging:
  level: "INFO"

Execute with:

python -m twinkle_eval.main --config my_config.yaml

Multilingual Batch Evaluation with Cloud Export

This configuration evaluates English and Chinese datasets with language-specific prompts and streams results to Google Sheets:

llm_api:
  base_url: "https://api.openai.com/v1"
  api_key: "${OPENAI_API_KEY}"
  type: "openai"

model:
  name: "gpt-4"
  temperature: 0.1

evaluation:
  dataset_paths:
    - "datasets/mmlu/"
    - "datasets/chinese/"
  evaluation_method: "pattern"
  system_prompt:
    en: "Answer with the correct option letter only."
    zh: "仅输出正确选项的字母。"
  datasets_prompt_map:
    "datasets/mmlu/": "en"
    "datasets/chinese/": "zh"
  repeat_runs: 1
  shuffle_options: true

google_services:
  google_sheets:
    enabled: true
    auth_method: "service_account"
    credentials_file: "service_account.json"
    spreadsheet_id: "1A2B3C4D5E"
    sheet_name: "MultilingualEval"

logging:
  level: "DEBUG"

Summary

• The default config template at twinkle_eval/config.template.yaml defines six primary sections: llm_api, model, evaluation, environment, logging, and google_services.
• The runtime configuration is built by ConfigurationManager.load_config(), which validates syntax, applies defaults via _apply_defaults() (lines 65-108), and verifies dataset paths through DatasetValidator.
• Customization focuses on the evaluation section for datasets, prompts, and methods, while llm_api and model control inference behavior.
• Google integrations are optional and configured under google_services for automatic export to Drive or Sheets.
• Factory classes (LLMFactory, EvaluationStrategyFactory) instantiate components based on the resolved configuration, supporting methods like box and pattern.

Frequently Asked Questions

What happens if I omit optional keys in my config file?

The ConfigurationManager automatically injects sensible defaults through the _apply_defaults() method in twinkle_eval/config.py. For example, if you omit llm_api.type, it defaults to "openai"; if you omit repeat_runs, it defaults to 1. The effective configuration always contains every key required for execution.

How do I validate my configuration before running an evaluation?

The framework validates your file automatically during the loading phase. validate_yaml_syntax checks for parse errors, while validate_config_structure ensures required sections exist. Additionally, DatasetValidator verifies that all paths in evaluation.dataset_paths point to existing directories with valid files. Run the CLI with --config to trigger these checks immediately.

Can I use environment variables instead of hardcoding API keys?

While the raw template shows hardcoded values for clarity, you can use shell environment variable substitution before passing the file to the framework, or modify the ConfigurationManager in twinkle_eval/config.py to resolve ${VAR} syntax. The google_services sections explicitly expect file paths for credentials (e.g., credentials_file: "service_account.json"), keeping secrets out of the main config.

What evaluation methods are supported and how do I switch between them?

The framework supports two primary methods: box (LaTeX-style \box{} wrapping) and pattern (regex-based extraction). Set evaluation.evaluation_method to either value. The EvaluationStrategyFactory.create_strategy method in twinkle_eval/evaluation_strategies.py instantiates the appropriate parser based on this key.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}