# Understanding Twinkle-Eval Result JSON Structure: A Complete Parsing Guide

> Parse Twinkle-Eval results JSON with our guide. Understand summary JSON and per-run JSONL files for AI model evaluation accuracy statistics and granular data. Get complete insights into your evaluation.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**Twinkle-Eval generates a single summary JSON file containing a timestamp, configuration metadata, and nested dataset_results with per-file accuracy statistics, plus optional per-run JSONL files for granular question-level data.**

The ai-twinkle/eval framework produces structured output files that capture every aspect of LLM evaluation runs. Understanding the exact schema of these **Twinkle-Eval result JSON files** is essential for building automated analysis pipelines, CI validation scripts, and custom reporting dashboards. This guide breaks down the file structure as implemented in the source code and demonstrates how to parse these files programmatically in Python.

## Anatomy of the Twinkle-Eval Results JSON

The primary output is a single summary file following the default naming pattern `results_<timestamp>.json`. According to the ai-twinkle/eval source code, this file is generated by `JSONExporter.export()` in [`twinkle_eval/results_exporters.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/results_exporters.py)【1†L30-L45】 and its content is assembled during `TwinkleEval.run_evaluation()` in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py)【2†L311-L316】.

### Top-Level Schema

The root JSON object contains four mandatory keys:

- **timestamp**: ISO-formatted string recording when the evaluation started (derived from `self.start_time`)
- **config**: The cleaned configuration object used for the run, including an injected **environment** sub-object containing GPU specifications, parallel-processing settings, and system metadata【1†L46-L70】
- **dataset_results**: A mapping where keys are dataset paths and values are dataset-level summary objects
- **duration_seconds**: Total wall-clock time in seconds for the complete evaluation run

### Dataset-Level Structure

Each entry under `dataset_results[<path>]` follows a dictionary structure built in `_evaluate_dataset()`【3†L55-L59】:

```json
{
  "results": [],
  "average_accuracy": 0.87,
  "average_std": 0.03
}

```

The fields include:
- **results**: An array of file-level result objects (see below)
- **average_accuracy**: The mean accuracy across all evaluated files within this specific dataset
- **average_std**: The standard deviation of accuracies across those files

### File-Level Results and Individual Runs

Every element in the `results` array represents a single evaluated data file and contains:

- **file**: Absolute or relative path to the evaluated data file
- **accuracy_mean**: The mean accuracy computed over the configured number of repeats (`repeat_runs`)
- **accuracy_std**: The standard deviation of accuracy across those repeat runs
- **individual_runs**: A nested object containing detailed per-run data:
  - **accuracies**: A list of accuracy scores, one for each repeat run
  - **results**: A list of file paths pointing to per-run **JSONL** files containing detailed question-level logs

### Per-Run JSONL Detail Files

For granular inspection, `Evaluator.evaluate_file()` in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py)【4†L40-L45】 writes line-delimited JSON files (typically named `eval_results_<timestamp>.jsonl`). Each line in these files represents a single question's complete evaluation record, including the question text, LLM output, and token usage statistics. The summary JSON references these files via the `individual_runs.results` array.

## Parsing Twinkle-Eval Results Programmatically

### Loading the Summary JSON

Use Python's standard `json` module to load the main results file and extract hierarchical statistics:

```python
import json
from pathlib import Path

# Load the summary file generated by Twinkle-Eval

summary_path = Path("results/results_20240101_1200.json")
with summary_path.open(encoding="utf-8") as f:
    data = json.load(f)

# Extract top-level metadata

run_timestamp = data["timestamp"]
config = data["config"]
duration = data["duration_seconds"]

# Iterate through each dataset's results

for dataset_path, ds_summary in data["dataset_results"].items():
    print(f"\nDataset: {dataset_path}")
    print(f"  Avg. accuracy: {ds_summary['average_accuracy']:.2%}")
    print(f"  Std. dev.:    {ds_summary['average_std']:.2%}")

    # Access individual file results within the dataset

    for file_res in ds_summary["results"]:
        print(f"  • File: {file_res['file']}")
        print(f"    Mean acc.: {file_res['accuracy_mean']:.2%}")
        print(f"    Std.:      {file_res['accuracy_std']:.2%}")

        # Access per-run details

        runs = file_res["individual_runs"]
        for i, acc in enumerate(runs["accuracies"], start=1):
            jsonl_path = runs["results"][i-1]
            print(f"      Run {i}: acc={acc:.2%}, details={jsonl_path}")

```

### Reading Per-Run JSONL Files

To analyze individual question-level results, parse the line-delimited JSON files referenced in the summary:

```python
def read_jsonl(path: Path):
    """Generator that yields JSON objects from a JSONL file."""
    with path.open(encoding="utf-8") as f:
        for line in f:
            yield json.loads(line)

# Example: Access the first run of the first file in the first dataset

first_dataset_key = list(data["dataset_results"])[0]
first_file = data["dataset_results"][first_dataset_key]["results"][0]
first_run_path = Path(first_file["individual_runs"]["results"][0])

# Iterate through individual question records

for record in read_jsonl(first_run_path):
    print(record["question_id"], record["is_correct"], record["usage_total_tokens"])

```

This approach allows you to programmatically extract overall run metadata, per-dataset statistics, per-file accuracy distributions, and granular per-question logs for deep-dive analysis.

## Summary

- The **summary JSON file** (`results_<timestamp>.json`) serves as the central artifact containing complete evaluation metadata, configuration, and nested results hierarchies.
- **Dataset-level summaries** aggregate statistics across files, providing `average_accuracy` and `average_std` for quick comparison.
- **File-level objects** contain `accuracy_mean` and `accuracy_std` across repeat runs, with `individual_runs` linking to detailed JSONL logs.
- **Per-run JSONL files** contain line-delimited question-level details written by `Evaluator.evaluate_file()`, enabling granular inspection of specific LLM responses.
- All structures are defined in [`twinkle_eval/results_exporters.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/results_exporters.py), [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py), and [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py).

## Frequently Asked Questions

### What is the default naming convention for Twinkle-Eval result files?

The framework generates summary files using the pattern `results_<timestamp>.json`, where the timestamp corresponds to the evaluation start time. Per-run detail files use the pattern `eval_results_<timestamp>.jsonl`. These naming conventions are enforced in `JSONExporter.export()` and `Evaluator.evaluate_file()` respectively.

### How do I access per-question evaluation details programmatically?

Per-question details are stored in JSONL files (one per repeat run) referenced via the `individual_runs.results` array within each file-level result object. Parse these files line-by-line using `json.loads()` to access fields like `question_id`, `is_correct`, and `usage_total_tokens` for each individual evaluation.

### Where in the source code is the results JSON structure defined?

The structure is assembled across three key locations: [`twinkle_eval/results_exporters.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/results_exporters.py) defines the `JSONExporter` class that builds the final output【1†L30-L70】; [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) contains `TwinkleEval.run_evaluation()` which aggregates the data【2†L311-L316】 and `_evaluate_dataset()` which creates the per-dataset summary objects【3†L55-L59】; and [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) handles the per-run JSONL generation【4†L40-L45】.

### Can I customize the environment metadata in the config section?

Yes. The `JSONExporter.export()` method automatically injects an `environment` sub-object into the `config` section containing GPU, parallel-processing, and system information【1†L46-L70】. While this is populated automatically, you can extend the exporter class to inject additional custom metadata fields if you modify the source in [`twinkle_eval/results_exporters.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/results_exporters.py).