How to Add Support for a New Dataset Format in Twinkle Eval: A Complete Guide

Adding support for a new dataset format in Twinkle Eval requires modifying three specific locations in twinkle_eval/dataset.py: detecting the file extension in the _load_data method, implementing the parsing logic, and registering the extension in the supported_extensions set.

Twinkle Eval is the evaluation framework within the ai-twinkle/eval repository designed to standardize how benchmark datasets are loaded and processed. When you need to integrate a proprietary or emerging data format—whether it is Excel, XML, or a custom binary format—the framework provides a clear extension point through the Dataset class. Understanding this process ensures your evaluation pipelines remain compatible with diverse data sources without forking the core library.

Understanding the Dataset Loading Architecture

The Dataset class in twinkle_eval/dataset.py serves as the central abstraction for data ingestion. It exposes a private method _load_data (starting at line 42) that inspects file extensions and dispatches to the appropriate parser. This method handles existing formats like .json, .csv, and .parquet through a series of conditional branches.

Parallel to the class instance method, the module-level function find_all_evaluation_files performs directory traversal to auto-discover evaluation datasets. This function relies on a module-level set named supported_extensions (defined at lines 107-114) to filter relevant files. Both components must be updated in tandem to fully integrate a new format.

Step-by-Step Process for Adding a New Dataset Format

Step 1: Detect the File Extension in _load_data

Locate the _load_data method in twinkle_eval/dataset.py around line 42. Inside this method, the code extracts the file extension using ext = Path(self.file_path).suffix.lower(). You must add a new conditional branch to recognize your target extension.

Insert an elif statement after the existing extension checks. For example, if adding support for Excel files, you would add:

elif ext == ".xlsx":
    # Loading logic will go here

    pass

Step 2: Implement the Loading Routine

Immediately following your new extension check, implement the parsing logic that converts the raw file into a list of dictionaries. Each dictionary must contain at minimum question and answer keys to maintain compatibility with the evaluation engine.

For tabular formats, pandas is the recommended library. For JSON-based formats, use the standard json module. Place your implementation after the existing elif ext in [".csv", ".tsv"]: block (around lines 65-78).

Your implementation should validate required columns, normalize data types, and handle edge cases like missing values. The data structure must match the format used by existing loaders:

data = [
    {"question": "What is 2+2?", "answer": "4"},
    {"question": "Capital of France?", "answer": "Paris"}
]

Step 3: Register the Extension in supported_extensions

Navigate to lines 107-114 in twinkle_eval/dataset.py to find the supported_extensions set. This set controls which files the find_all_evaluation_files function discovers when scanning directories.

Add your new extension string to this set. For example:

supported_extensions = {
    ".json",
    ".jsonl",
    ".parquet",
    ".arrow",
    ".csv",
    ".tsv",
    ".xlsx",  # Newly added

}

Without this registration, the CLI will ignore files of your new type even though the Dataset class can parse them.

Practical Example: Adding Excel (.xlsx) Support

Below is a complete implementation example demonstrating how to add Excel support to Twinkle Eval. This example assumes you have added openpyxl to your project dependencies.

First, modify the _load_data method in twinkle_eval/dataset.py:

elif ext == ".xlsx":
    import pandas as pd
    
    df = pd.read_excel(self.file_path, engine="openpyxl")
    
    # Validate required columns exist

    required_cols = ["question", "answer"]
    missing = [col for col in required_cols if col not in df.columns]
    if missing:
        raise ValueError(
            f"Missing required columns {missing} in `{self.file_path}`"
        )
    
    # Normalize answer column to match existing format expectations

    df["answer"] = df["answer"].astype(str).str.strip().str.upper()
    
    # Convert to list of dictionaries

    self.data = df.to_dict(orient="records")

Then register the extension at lines 107-114:

supported_extensions = {
    ".json",
    ".jsonl", 
    ".parquet",
    ".arrow",
    ".csv",
    ".tsv",
    ".xlsx",  # Excel support added

}

Finally, add the dependency to pyproject.toml:

[project.dependencies]
pandas = ">=2.0.0"
openpyxl = ">=3.1.0"  # Required for Excel support

Testing Your New Dataset Format

After implementing the changes, verify your integration using both the CLI and programmatic APIs.

CLI Verification:

twinkle-eval evaluate /path/to/test_data.xlsx \
    --config config.yaml \
    --output results.json

Programmatic Verification:

from twinkle_eval.dataset import Dataset, find_all_evaluation_files

# Test single file loading

ds = Dataset("benchmarks/new_format.xlsx")
assert len(ds) > 0
assert "question" in ds[0]
assert "answer" in ds[0]

# Test directory discovery

files = find_all_evaluation_files("benchmarks/")
assert any(str(f).endswith(".xlsx") for f in files)

Run the existing test suite to ensure backward compatibility:

pytest tests/test_dataset.py -v

Summary

Adding support for a new dataset format in Twinkle Eval involves a systematic three-step modification to twinkle_eval/dataset.py:

• Extend _load_data to detect the new file extension and dispatch to your parser
• Implement the loading logic that converts raw files into standardized {question, answer} dictionaries using pandas or standard library modules
• Update supported_extensions to enable automatic file discovery by find_all_evaluation_files

Always validate required columns, normalize data types to match existing format expectations, and add corresponding dependencies to pyproject.toml. Test your implementation using both the CLI and Python API to ensure seamless integration with the existing evaluation pipeline.

Frequently Asked Questions

What dependencies are required when adding support for binary formats like Excel?

Binary formats typically require additional Python libraries beyond the core Twinkle Eval dependencies. For Excel files, you must install openpyxl (for .xlsx) or xlrd (for older .xls files). Add these to the [project.dependencies] section of pyproject.toml and document them in your README.md. Always import these libraries inside the specific elif branch rather than at the module level to keep startup times fast for users who do not need that format.

Can I add support for multiple new formats in a single pull request?

Yes, you can implement multiple format handlers within the same modification to twinkle_eval/dataset.py. Each format requires its own extension detection branch in _load_data and its own entry in the supported_extensions set. However, ensure each format has independent test coverage and that you do not mix unrelated logic. If the formats require different heavy dependencies (e.g., one needs pyarrow for Parquet and another needs openpyxl for Excel), consider lazy imports to avoid forcing all dependencies on all users.

How does Twinkle Eval validate that my new format returns the correct data structure?

The Dataset class expects the internal self.data attribute to be a list of dictionaries, where each dictionary contains at least question and answer keys. During your implementation, you should explicitly validate that these columns exist after parsing, raising a ValueError with a descriptive message if they are missing. The evaluation engine iterates over this list and passes each record to the model for inference, so any deviation from this structure will raise KeyError exceptions during the evaluation phase. Always test your implementation with a small sample file before running large-scale evaluations.

Is it possible to add support for remote or cloud-hosted datasets?

The current implementation in twinkle_eval/dataset.py assumes local file paths, but you can extend the _load_data method to handle remote protocols. Add a conditional check at the beginning of the method to detect URL schemes (e.g., http://, s3://, or gs://). If detected, use appropriate libraries like requests, boto3, or gcsfs to download the file to a temporary location, then proceed with the standard extension-based parsing logic. Remember to clean up temporary files and add the required cloud dependencies to pyproject.toml as optional extras.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}