Understanding Twinkle-Eval Result JSON Structure: A Complete Parsing Guide

Twinkle-Eval generates a single summary JSON file containing a timestamp, configuration metadata, and nested dataset_results with per-file accuracy statistics, plus optional per-run JSONL files for granular question-level data.

The ai-twinkle/eval framework produces structured output files that capture every aspect of LLM evaluation runs. Understanding the exact schema of these Twinkle-Eval result JSON files is essential for building automated analysis pipelines, CI validation scripts, and custom reporting dashboards. This guide breaks down the file structure as implemented in the source code and demonstrates how to parse these files programmatically in Python.

Anatomy of the Twinkle-Eval Results JSON

The primary output is a single summary file following the default naming pattern results_<timestamp>.json. According to the ai-twinkle/eval source code, this file is generated by JSONExporter.export() in twinkle_eval/results_exporters.py【1†L30-L45】 and its content is assembled during TwinkleEval.run_evaluation() in twinkle_eval/main.py【2†L311-L316】.

Top-Level Schema

The root JSON object contains four mandatory keys:

• timestamp: ISO-formatted string recording when the evaluation started (derived from self.start_time)
• config: The cleaned configuration object used for the run, including an injected environment sub-object containing GPU specifications, parallel-processing settings, and system metadata【1†L46-L70】
• dataset_results: A mapping where keys are dataset paths and values are dataset-level summary objects
• duration_seconds: Total wall-clock time in seconds for the complete evaluation run

Dataset-Level Structure

Each entry under dataset_results[<path>] follows a dictionary structure built in _evaluate_dataset()【3†L55-L59】:

{
  "results": [],
  "average_accuracy": 0.87,
  "average_std": 0.03
}

The fields include:

• results: An array of file-level result objects (see below)
• average_accuracy: The mean accuracy across all evaluated files within this specific dataset
• average_std: The standard deviation of accuracies across those files

File-Level Results and Individual Runs

Every element in the results array represents a single evaluated data file and contains:

• file: Absolute or relative path to the evaluated data file
• accuracy_mean: The mean accuracy computed over the configured number of repeats (repeat_runs)
• accuracy_std: The standard deviation of accuracy across those repeat runs
• individual_runs: A nested object containing detailed per-run data:
  • accuracies: A list of accuracy scores, one for each repeat run
  • results: A list of file paths pointing to per-run JSONL files containing detailed question-level logs

Per-Run JSONL Detail Files

For granular inspection, Evaluator.evaluate_file() in twinkle_eval/evaluators.py【4†L40-L45】 writes line-delimited JSON files (typically named eval_results_<timestamp>.jsonl). Each line in these files represents a single question's complete evaluation record, including the question text, LLM output, and token usage statistics. The summary JSON references these files via the individual_runs.results array.

Parsing Twinkle-Eval Results Programmatically

Loading the Summary JSON

Use Python's standard json module to load the main results file and extract hierarchical statistics:

import json
from pathlib import Path

# Load the summary file generated by Twinkle-Eval

summary_path = Path("results/results_20240101_1200.json")
with summary_path.open(encoding="utf-8") as f:
    data = json.load(f)

# Extract top-level metadata

run_timestamp = data["timestamp"]
config = data["config"]
duration = data["duration_seconds"]

# Iterate through each dataset's results

for dataset_path, ds_summary in data["dataset_results"].items():
    print(f"\nDataset: {dataset_path}")
    print(f"  Avg. accuracy: {ds_summary['average_accuracy']:.2%}")
    print(f"  Std. dev.:    {ds_summary['average_std']:.2%}")

    # Access individual file results within the dataset

    for file_res in ds_summary["results"]:
        print(f"  • File: {file_res['file']}")
        print(f"    Mean acc.: {file_res['accuracy_mean']:.2%}")
        print(f"    Std.:      {file_res['accuracy_std']:.2%}")

        # Access per-run details

        runs = file_res["individual_runs"]
        for i, acc in enumerate(runs["accuracies"], start=1):
            jsonl_path = runs["results"][i-1]
            print(f"      Run {i}: acc={acc:.2%}, details={jsonl_path}")

Reading Per-Run JSONL Files

To analyze individual question-level results, parse the line-delimited JSON files referenced in the summary:

def read_jsonl(path: Path):
    """Generator that yields JSON objects from a JSONL file."""
    with path.open(encoding="utf-8") as f:
        for line in f:
            yield json.loads(line)

# Example: Access the first run of the first file in the first dataset

first_dataset_key = list(data["dataset_results"])[0]
first_file = data["dataset_results"][first_dataset_key]["results"][0]
first_run_path = Path(first_file["individual_runs"]["results"][0])

# Iterate through individual question records

for record in read_jsonl(first_run_path):
    print(record["question_id"], record["is_correct"], record["usage_total_tokens"])

This approach allows you to programmatically extract overall run metadata, per-dataset statistics, per-file accuracy distributions, and granular per-question logs for deep-dive analysis.

Summary

• The summary JSON file (results_<timestamp>.json) serves as the central artifact containing complete evaluation metadata, configuration, and nested results hierarchies.
• Dataset-level summaries aggregate statistics across files, providing average_accuracy and average_std for quick comparison.
• File-level objects contain accuracy_mean and accuracy_std across repeat runs, with individual_runs linking to detailed JSONL logs.
• Per-run JSONL files contain line-delimited question-level details written by Evaluator.evaluate_file(), enabling granular inspection of specific LLM responses.
• All structures are defined in twinkle_eval/results_exporters.py, twinkle_eval/main.py, and twinkle_eval/evaluators.py.

Frequently Asked Questions

What is the default naming convention for Twinkle-Eval result files?

The framework generates summary files using the pattern results_<timestamp>.json, where the timestamp corresponds to the evaluation start time. Per-run detail files use the pattern eval_results_<timestamp>.jsonl. These naming conventions are enforced in JSONExporter.export() and Evaluator.evaluate_file() respectively.

How do I access per-question evaluation details programmatically?

Per-question details are stored in JSONL files (one per repeat run) referenced via the individual_runs.results array within each file-level result object. Parse these files line-by-line using json.loads() to access fields like question_id, is_correct, and usage_total_tokens for each individual evaluation.

Where in the source code is the results JSON structure defined?

The structure is assembled across three key locations: twinkle_eval/results_exporters.py defines the JSONExporter class that builds the final output【1†L30-L70】; twinkle_eval/main.py contains TwinkleEval.run_evaluation() which aggregates the data【2†L311-L316】 and _evaluate_dataset() which creates the per-dataset summary objects【3†L55-L59】; and twinkle_eval/evaluators.py handles the per-run JSONL generation【4†L40-L45】.

Can I customize the environment metadata in the config section?

Yes. The JSONExporter.export() method automatically injects an environment sub-object into the config section containing GPU, parallel-processing, and system information【1†L46-L70】. While this is populated automatically, you can extend the exporter class to inject additional custom metadata fields if you modify the source in twinkle_eval/results_exporters.py.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}