How to Debug Evaluation Failures Using the Logging Configuration in Twinkle Eval

To debug evaluation failures in Twinkle Eval, inspect the timestamped log files in the logs/ directory, which capture chronological execution traces, error messages, and component-specific diagnostics generated by the built-in log_info, log_error, and log_warning helpers.

Twinkle Eval, the open-source evaluation framework in the ai-twinkle/eval repository, ships with a built-in logging system that automatically records every step of an evaluation run. By configuring and reading these logs, you can quickly pinpoint exactly which configuration file, dataset, or API call caused a failure without manually instrumenting your code.

Understanding Twinkle Eval's Built-In Logging System

The logging infrastructure is centralized in twinkle_eval/logger.py and consumed by all major components including the configuration loader, dataset reader, and benchmark runner.

Log File Generation and Structure

Every evaluation run generates a unique log file based on a timestamp:

1. Directory creation – The system ensures a logs/ folder exists via os.makedirs(logs_dir, exist_ok=True) in logger.py lines 9‑11.
2. Timestamp generation – A string formatted as %Y%m%d_%H%M is created using datetime.now() in logger.py lines 6‑8.
3. File configuration – The root logger writes to logs/evaluation_<timestamp>.log with INFO level and a simple format string, as configured in logger.py lines 13‑21.

Helper Functions and Log Levels

Instead of calling the standard library directly, components use wrapper functions defined in logger.py lines 24‑34:

• log_info(message) – General progress updates.
• log_warning(message) – Non-fatal issues.
• log_error(message) – Critical failures that halt or corrupt a specific evaluation step.

These wrappers ensure consistent formatting and encoding (UTF‑8) across all output.

Locating and Reading Log Files

After a run completes—or crashes—find the most recent log file using standard shell tools:

# List the newest log file

ls -1t logs/evaluation_*.log | head -n1

# → logs/evaluation_20260223_1542.log

Search for error entries to jump directly to the failure point:

grep -i "error" logs/evaluation_20260223_1542.log

Example output showing a dataset parsing failure:

2026-02-23 15:42:07,123 - ERROR - 評測檔案 /data/mmlu/physics.json 失敗: KeyError('answer')

Debugging Common Evaluation Failures

Each major pipeline stage emits specific log messages that help isolate problems.

Configuration Loading Errors

In twinkle_eval/config.py lines 55‑60, the configuration loader wraps validation in a try‑except block:

• Success: log_info("配置載入和驗證完成")
• Failure: log_error(f"配置錯誤: {e}")

If you see a configuration error in the log, verify that your config.yaml follows the expected schema and that all referenced paths exist.

Dataset Parsing Failures

The dataset reader in twinkle_eval/dataset.py lines 79‑83 logs both successful loads and parsing exceptions:

• Success: log_info(f"成功讀取檔案: {self.file_path}，共 {len(data)} 題")
• Failure: log_error(f"讀取資料錯誤: {e}")

A KeyError or JSONDecodeError here usually indicates malformed input data or a mismatch between the expected and actual JSON schema.

Runtime Evaluation Errors

During the main execution loop in twinkle_eval/main.py lines 99‑102, the runner catches per‑file exceptions:

log_info(f"開始評測資料集: {dataset_path}")

# ... evaluation logic ...

log_error(f"評測檔案 {file_path} 失敗: {e}")

Benchmark-specific errors are handled similarly in main.py lines 45‑48 with log_error(f"基準測試執行錯誤: {e}").

Customizing Logging for Advanced Debugging

For deeper introspection, modify twinkle_eval/logger.py before running your evaluation.

Enable Debug-Level Verbosity

Change the level parameter in the logging.basicConfig call (lines 13‑21) from logging.INFO to logging.DEBUG:

logging.basicConfig(
    filename=log_filename,
    level=logging.DEBUG,      # <- increase verbosity

    format="%(asctime)s - %(levelname)s - %(message)s",
    encoding="utf-8",
)

Now add granular debug statements in your own extensions:

import logging
logging.debug(f"Evaluating file {file_path} with prompt language {dataset_lang}")

Add Console Output

To see logs in real-time without opening the file, add a StreamHandler to the logger configuration:

console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
logging.getLogger().addHandler(console_handler)

Adjust the Log Format

Include module names or line numbers by updating the format string in logger.py:

format="%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(message)s"

Uploading Logs to Google Drive

Twinkle Eval can automatically push log files to Google Drive for centralized debugging, implemented in logger.py lines 36‑70.

Configuration

Enable the feature in your config.yaml:

google_services:
  google_drive:
    enabled: true
    credentials_path: "gdrive_service_account.json"
    folder_name: "twinkle-eval-logs"

When enabled: true, the runner calls upload_logs_to_drive() after evaluation completes. Success is confirmed via log_info messages in main.py lines 50‑56:

成功上傳 3 個檔案到 Google Drive

This allows team members to debug failures without direct access to the local machine.

Summary

• Twinkle Eval generates timestamped log files in the logs/ directory for every run, configured in twinkle_eval/logger.py.
• Three helper functions—log_info, log_error, and log_warning—standardize output across the configuration loader, dataset reader, and evaluation runner.
• Error tracing is straightforward: grep the log file for ERROR entries to find the exact file path and exception message emitted by components like config.py, dataset.py, or main.py.
• Customization involves editing logger.py to set level=logging.DEBUG, add StreamHandler for console output, or modify the format string.
• Google Drive integration automatically uploads logs when enabled in config.yaml, facilitating remote debugging of evaluation failures.

Frequently Asked Questions

Where are Twinkle Eval log files stored?

Log files are stored in the logs/ directory relative to the execution path. Each run creates a uniquely named file following the pattern evaluation_<timestamp>.log, where the timestamp uses the format %Y%m%d_%H%M (e.g., evaluation_20260223_1542.log). This is handled in twinkle_eval/logger.py lines 6‑11.

How do I enable debug-level logging for more verbose output?

To enable DEBUG level logging, modify twinkle_eval/logger.py and change the level parameter in the logging.basicConfig call from logging.INFO to logging.DEBUG (lines 13‑21). You can then add logging.debug() statements in your custom code to capture granular execution details. Alternatively, add a StreamHandler to see debug output in the console in real-time.

Can Twinkle Eval automatically upload logs to Google Drive?

Yes. When you set google_services.google_drive.enabled: true in your config.yaml and provide a valid credentials_path, Twinkle Eval automatically calls upload_logs_to_drive() from logger.py after the evaluation completes. This uploads all files from the logs/ directory to the specified Google Drive folder, allowing remote debugging of evaluation failures without SSH access to the host machine.

What should I look for in the log file when an evaluation fails?

Search for entries marked with ERROR or WARNING. The log uses structured prefixes like 2026-02-23 15:42:07,123 - ERROR - followed by descriptive messages such as 評測檔案 /path/to/file.json 失敗: KeyError('answer'). Cross-reference the file path mentioned in the error with the source code locations in config.py, dataset.py, or main.py to identify the exact failure point.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}