# How to Debug Evaluation Failures Using the Logging Configuration in Twinkle Eval

> Debug Twinkle Eval evaluation failures by inspecting timestamped log files in the logs directory. Utilize log_info, log_error, and log_warning helpers for detailed diagnostics.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**To debug evaluation failures in Twinkle Eval, inspect the timestamped log files in the `logs/` directory, which capture chronological execution traces, error messages, and component-specific diagnostics generated by the built-in `log_info`, `log_error`, and `log_warning` helpers.**

Twinkle Eval, the open-source evaluation framework in the `ai-twinkle/eval` repository, ships with a built-in logging system that automatically records every step of an evaluation run. By configuring and reading these logs, you can quickly pinpoint exactly which configuration file, dataset, or API call caused a failure without manually instrumenting your code.

## Understanding Twinkle Eval's Built-In Logging System

The logging infrastructure is centralized in [`twinkle_eval/logger.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/logger.py) and consumed by all major components including the configuration loader, dataset reader, and benchmark runner.

### Log File Generation and Structure

Every evaluation run generates a unique log file based on a timestamp:

1. **Directory creation** – The system ensures a `logs/` folder exists via `os.makedirs(logs_dir, exist_ok=True)` in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) lines 9‑11.
2. **Timestamp generation** – A string formatted as `%Y%m%d_%H%M` is created using `datetime.now()` in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) lines 6‑8.
3. **File configuration** – The root logger writes to `logs/evaluation_<timestamp>.log` with **INFO** level and a simple format string, as configured in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) lines 13‑21.

### Helper Functions and Log Levels

Instead of calling the standard library directly, components use wrapper functions defined in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) lines 24‑34:

- `log_info(message)` – General progress updates.
- `log_warning(message)` – Non-fatal issues.
- `log_error(message)` – Critical failures that halt or corrupt a specific evaluation step.

These wrappers ensure consistent formatting and encoding (UTF‑8) across all output.

## Locating and Reading Log Files

After a run completes—or crashes—find the most recent log file using standard shell tools:

```bash

# List the newest log file

ls -1t logs/evaluation_*.log | head -n1

# → logs/evaluation_20260223_1542.log

```

Search for error entries to jump directly to the failure point:

```bash
grep -i "error" logs/evaluation_20260223_1542.log

```

Example output showing a dataset parsing failure:

```

2026-02-23 15:42:07,123 - ERROR - 評測檔案 /data/mmlu/physics.json 失敗: KeyError('answer')

```

## Debugging Common Evaluation Failures

Each major pipeline stage emits specific log messages that help isolate problems.

### Configuration Loading Errors

In [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) lines 55‑60, the configuration loader wraps validation in a try‑except block:

- **Success**: `log_info("配置載入和驗證完成")`
- **Failure**: `log_error(f"配置錯誤: {e}")`

If you see a configuration error in the log, verify that your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) follows the expected schema and that all referenced paths exist.

### Dataset Parsing Failures

The dataset reader in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py) lines 79‑83 logs both successful loads and parsing exceptions:

- **Success**: `log_info(f"成功讀取檔案: {self.file_path}，共 {len(data)} 題")`
- **Failure**: `log_error(f"讀取資料錯誤: {e}")`

A `KeyError` or `JSONDecodeError` here usually indicates malformed input data or a mismatch between the expected and actual JSON schema.

### Runtime Evaluation Errors

During the main execution loop in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) lines 99‑102, the runner catches per‑file exceptions:

```python
log_info(f"開始評測資料集: {dataset_path}")

# ... evaluation logic ...

log_error(f"評測檔案 {file_path} 失敗: {e}")

```

Benchmark-specific errors are handled similarly in [`main.py`](https://github.com/ai-twinkle/eval/blob/main/main.py) lines 45‑48 with `log_error(f"基準測試執行錯誤: {e}")`.

## Customizing Logging for Advanced Debugging

For deeper introspection, modify [`twinkle_eval/logger.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/logger.py) before running your evaluation.

### Enable Debug-Level Verbosity

Change the `level` parameter in the `logging.basicConfig` call (lines 13‑21) from `logging.INFO` to `logging.DEBUG`:

```python
logging.basicConfig(
    filename=log_filename,
    level=logging.DEBUG,      # <- increase verbosity

    format="%(asctime)s - %(levelname)s - %(message)s",
    encoding="utf-8",
)

```

Now add granular debug statements in your own extensions:

```python
import logging
logging.debug(f"Evaluating file {file_path} with prompt language {dataset_lang}")

```

### Add Console Output

To see logs in real-time without opening the file, add a `StreamHandler` to the logger configuration:

```python
console_handler = logging.StreamHandler()
console_handler.setLevel(logging.INFO)
logging.getLogger().addHandler(console_handler)

```

### Adjust the Log Format

Include module names or line numbers by updating the format string in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py):

```python
format="%(asctime)s - %(name)s - %(lineno)d - %(levelname)s - %(message)s"

```

## Uploading Logs to Google Drive

Twinkle Eval can automatically push log files to Google Drive for centralized debugging, implemented in [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) lines 36‑70.

### Configuration

Enable the feature in your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml):

```yaml
google_services:
  google_drive:
    enabled: true
    credentials_path: "gdrive_service_account.json"
    folder_name: "twinkle-eval-logs"

```

When `enabled: true`, the runner calls `upload_logs_to_drive()` after evaluation completes. Success is confirmed via `log_info` messages in [`main.py`](https://github.com/ai-twinkle/eval/blob/main/main.py) lines 50‑56:

```

成功上傳 3 個檔案到 Google Drive

```

This allows team members to debug failures without direct access to the local machine.

## Summary

- **Twinkle Eval** generates timestamped log files in the `logs/` directory for every run, configured in [`twinkle_eval/logger.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/logger.py).
- **Three helper functions**—`log_info`, `log_error`, and `log_warning`—standardize output across the configuration loader, dataset reader, and evaluation runner.
- **Error tracing** is straightforward: grep the log file for `ERROR` entries to find the exact file path and exception message emitted by components like [`config.py`](https://github.com/ai-twinkle/eval/blob/main/config.py), [`dataset.py`](https://github.com/ai-twinkle/eval/blob/main/dataset.py), or [`main.py`](https://github.com/ai-twinkle/eval/blob/main/main.py).
- **Customization** involves editing [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) to set `level=logging.DEBUG`, add `StreamHandler` for console output, or modify the format string.
- **Google Drive integration** automatically uploads logs when enabled in [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml), facilitating remote debugging of evaluation failures.

## Frequently Asked Questions

### Where are Twinkle Eval log files stored?

Log files are stored in the `logs/` directory relative to the execution path. Each run creates a uniquely named file following the pattern `evaluation_<timestamp>.log`, where the timestamp uses the format `%Y%m%d_%H%M` (e.g., `evaluation_20260223_1542.log`). This is handled in [`twinkle_eval/logger.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/logger.py) lines 6‑11.

### How do I enable debug-level logging for more verbose output?

To enable **DEBUG** level logging, modify [`twinkle_eval/logger.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/logger.py) and change the `level` parameter in the `logging.basicConfig` call from `logging.INFO` to `logging.DEBUG` (lines 13‑21). You can then add `logging.debug()` statements in your custom code to capture granular execution details. Alternatively, add a `StreamHandler` to see debug output in the console in real-time.

### Can Twinkle Eval automatically upload logs to Google Drive?

Yes. When you set `google_services.google_drive.enabled: true` in your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) and provide a valid `credentials_path`, Twinkle Eval automatically calls `upload_logs_to_drive()` from [`logger.py`](https://github.com/ai-twinkle/eval/blob/main/logger.py) after the evaluation completes. This uploads all files from the `logs/` directory to the specified Google Drive folder, allowing remote debugging of evaluation failures without SSH access to the host machine.

### What should I look for in the log file when an evaluation fails?

Search for entries marked with `ERROR` or `WARNING`. The log uses structured prefixes like `2026-02-23 15:42:07,123 - ERROR -` followed by descriptive messages such as `評測檔案 /path/to/file.json 失敗: KeyError('answer')`. Cross-reference the file path mentioned in the error with the source code locations in [`config.py`](https://github.com/ai-twinkle/eval/blob/main/config.py), [`dataset.py`](https://github.com/ai-twinkle/eval/blob/main/dataset.py), or [`main.py`](https://github.com/ai-twinkle/eval/blob/main/main.py) to identify the exact failure point.