# How to Add Support for a New Dataset Format in Twinkle Eval: A Complete Guide

> Easily add support for new dataset formats in ai-twinkle/eval. Learn the simple three-step process to extend Twinkle Eval's capabilities with our complete guide.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**Adding support for a new dataset format in Twinkle Eval requires modifying three specific locations in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py): detecting the file extension in the `_load_data` method, implementing the parsing logic, and registering the extension in the `supported_extensions` set.**

Twinkle Eval is the evaluation framework within the `ai-twinkle/eval` repository designed to standardize how benchmark datasets are loaded and processed. When you need to integrate a proprietary or emerging data format—whether it is Excel, XML, or a custom binary format—the framework provides a clear extension point through the `Dataset` class. Understanding this process ensures your evaluation pipelines remain compatible with diverse data sources without forking the core library.

## Understanding the Dataset Loading Architecture

The `Dataset` class in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py) serves as the central abstraction for data ingestion. It exposes a private method `_load_data` (starting at line 42) that inspects file extensions and dispatches to the appropriate parser. This method handles existing formats like `.json`, `.csv`, and `.parquet` through a series of conditional branches.

Parallel to the class instance method, the module-level function `find_all_evaluation_files` performs directory traversal to auto-discover evaluation datasets. This function relies on a module-level set named `supported_extensions` (defined at lines 107-114) to filter relevant files. Both components must be updated in tandem to fully integrate a new format.

## Step-by-Step Process for Adding a New Dataset Format

### Step 1: Detect the File Extension in `_load_data`

Locate the `_load_data` method in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py) around line 42. Inside this method, the code extracts the file extension using `ext = Path(self.file_path).suffix.lower()`. You must add a new conditional branch to recognize your target extension.

Insert an `elif` statement after the existing extension checks. For example, if adding support for Excel files, you would add:

```python
elif ext == ".xlsx":
    # Loading logic will go here

    pass

```

### Step 2: Implement the Loading Routine

Immediately following your new extension check, implement the parsing logic that converts the raw file into a list of dictionaries. Each dictionary must contain at minimum `question` and `answer` keys to maintain compatibility with the evaluation engine.

For tabular formats, **pandas** is the recommended library. For JSON-based formats, use the standard `json` module. Place your implementation after the existing `elif ext in [".csv", ".tsv"]:` block (around lines 65-78).

Your implementation should validate required columns, normalize data types, and handle edge cases like missing values. The data structure must match the format used by existing loaders:

```python
data = [
    {"question": "What is 2+2?", "answer": "4"},
    {"question": "Capital of France?", "answer": "Paris"}
]

```

### Step 3: Register the Extension in `supported_extensions`

Navigate to lines 107-114 in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py) to find the `supported_extensions` set. This set controls which files the `find_all_evaluation_files` function discovers when scanning directories.

Add your new extension string to this set. For example:

```python
supported_extensions = {
    ".json",
    ".jsonl",
    ".parquet",
    ".arrow",
    ".csv",
    ".tsv",
    ".xlsx",  # Newly added

}

```

Without this registration, the CLI will ignore files of your new type even though the `Dataset` class can parse them.

## Practical Example: Adding Excel (.xlsx) Support

Below is a complete implementation example demonstrating how to add Excel support to Twinkle Eval. This example assumes you have added `openpyxl` to your project dependencies.

First, modify the `_load_data` method in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py):

```python
elif ext == ".xlsx":
    import pandas as pd
    
    df = pd.read_excel(self.file_path, engine="openpyxl")
    
    # Validate required columns exist

    required_cols = ["question", "answer"]
    missing = [col for col in required_cols if col not in df.columns]
    if missing:
        raise ValueError(
            f"Missing required columns {missing} in `{self.file_path}`"
        )
    
    # Normalize answer column to match existing format expectations

    df["answer"] = df["answer"].astype(str).str.strip().str.upper()
    
    # Convert to list of dictionaries

    self.data = df.to_dict(orient="records")

```

Then register the extension at lines 107-114:

```python
supported_extensions = {
    ".json",
    ".jsonl", 
    ".parquet",
    ".arrow",
    ".csv",
    ".tsv",
    ".xlsx",  # Excel support added

}

```

Finally, add the dependency to [`pyproject.toml`](https://github.com/ai-twinkle/eval/blob/main/pyproject.toml):

```toml
[project.dependencies]
pandas = ">=2.0.0"
openpyxl = ">=3.1.0"  # Required for Excel support

```

## Testing Your New Dataset Format

After implementing the changes, verify your integration using both the CLI and programmatic APIs.

**CLI Verification:**

```bash
twinkle-eval evaluate /path/to/test_data.xlsx \
    --config config.yaml \
    --output results.json

```

**Programmatic Verification:**

```python
from twinkle_eval.dataset import Dataset, find_all_evaluation_files

# Test single file loading

ds = Dataset("benchmarks/new_format.xlsx")
assert len(ds) > 0
assert "question" in ds[0]
assert "answer" in ds[0]

# Test directory discovery

files = find_all_evaluation_files("benchmarks/")
assert any(str(f).endswith(".xlsx") for f in files)

```

Run the existing test suite to ensure backward compatibility:

```bash
pytest tests/test_dataset.py -v

```

## Summary

Adding support for a new dataset format in Twinkle Eval involves a systematic three-step modification to [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py):

- **Extend `_load_data`** to detect the new file extension and dispatch to your parser
- **Implement the loading logic** that converts raw files into standardized `{question, answer}` dictionaries using **pandas** or standard library modules
- **Update `supported_extensions`** to enable automatic file discovery by `find_all_evaluation_files`

Always validate required columns, normalize data types to match existing format expectations, and add corresponding dependencies to [`pyproject.toml`](https://github.com/ai-twinkle/eval/blob/main/pyproject.toml). Test your implementation using both the CLI and Python API to ensure seamless integration with the existing evaluation pipeline.

## Frequently Asked Questions

### What dependencies are required when adding support for binary formats like Excel?

Binary formats typically require additional Python libraries beyond the core Twinkle Eval dependencies. For Excel files, you must install `openpyxl` (for `.xlsx`) or `xlrd` (for older `.xls` files). Add these to the `[project.dependencies]` section of [`pyproject.toml`](https://github.com/ai-twinkle/eval/blob/main/pyproject.toml) and document them in your [`README.md`](https://github.com/ai-twinkle/eval/blob/main/README.md). Always import these libraries inside the specific `elif` branch rather than at the module level to keep startup times fast for users who do not need that format.

### Can I add support for multiple new formats in a single pull request?

Yes, you can implement multiple format handlers within the same modification to [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py). Each format requires its own extension detection branch in `_load_data` and its own entry in the `supported_extensions` set. However, ensure each format has independent test coverage and that you do not mix unrelated logic. If the formats require different heavy dependencies (e.g., one needs `pyarrow` for Parquet and another needs `openpyxl` for Excel), consider lazy imports to avoid forcing all dependencies on all users.

### How does Twinkle Eval validate that my new format returns the correct data structure?

The `Dataset` class expects the internal `self.data` attribute to be a list of dictionaries, where each dictionary contains at least `question` and `answer` keys. During your implementation, you should explicitly validate that these columns exist after parsing, raising a `ValueError` with a descriptive message if they are missing. The evaluation engine iterates over this list and passes each record to the model for inference, so any deviation from this structure will raise `KeyError` exceptions during the evaluation phase. Always test your implementation with a small sample file before running large-scale evaluations.

### Is it possible to add support for remote or cloud-hosted datasets?

The current implementation in [`twinkle_eval/dataset.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/dataset.py) assumes local file paths, but you can extend the `_load_data` method to handle remote protocols. Add a conditional check at the beginning of the method to detect URL schemes (e.g., `http://`, `s3://`, or `gs://`). If detected, use appropriate libraries like `requests`, `boto3`, or `gcsfs` to download the file to a temporary location, then proceed with the standard extension-based parsing logic. Remember to clean up temporary files and add the required cloud dependencies to [`pyproject.toml`](https://github.com/ai-twinkle/eval/blob/main/pyproject.toml) as optional extras.