How to Use Twinkle Eval Programmatically with the Python API: A Complete Guide

To use Twinkle Eval programmatically, import TwinkleEvalRunner from the twinkle-eval package, instantiate it with a configuration file path, call initialize() to load settings, and execute run_evaluation() to process datasets and export results.

The ai-twinkle/eval repository provides a modular framework for evaluating large-language-model (LLM) responses against standardized datasets. While the CLI offers quick access, the Python API gives you fine-grained control over configuration loading, LLM instantiation, evaluation strategies, and result export formats. This guide demonstrates how to use Twinkle Eval programmatically using the core classes defined in the source code.

Installation and Prerequisites

Install the package from PyPI to access the programmatic interface:

pip install twinkle-eval

The setup.py in the repository publishes this package name, ensuring you receive the correct distribution with all dependencies.

Core Architecture Overview

Understanding the key classes helps you use Twinkle Eval programmatically with precision. The framework follows a factory pattern for extensibility:

• TwinkleEvalRunner (twinkle_eval/main.py, lines 61-84): The primary entry point that orchestrates the entire pipeline.
• ConfigurationManager / load_config (twinkle_eval/config.py): Parses YAML configuration and validates required sections.
• LLMFactory (twinkle_eval/models.py, lines 9-33): Creates LLM instances (OpenAI-compatible) with validated API credentials.
• EvaluationStrategyFactory (twinkle_eval/evaluation_strategies.py): Returns concrete strategies like BoxExtractionStrategy or PatternMatchingStrategy.
• Evaluator (twinkle_eval/evaluators.py, lines 31-48): Handles parallel execution, rate limiting, and answer extraction.
• ResultsExporterFactory (twinkle_eval/results_exporters.py): Converts summaries to JSON, CSV, HTML, or Google Sheets.

Basic Programmatic Usage

The standard workflow involves three steps: instantiate the runner, initialize internal state, and execute the evaluation.

Loading Configuration

First, create a YAML configuration file that specifies your LLM API details, model parameters, and evaluation method:

llm_api:
  base_url: "http://localhost:8000/v1"
  api_key: "YOUR_API_KEY"
  api_rate_limit: 2
model:
  name: "gpt-4o-mini"
  temperature: 0.0
  top_p: 0.9
evaluation:
  dataset_paths: ["datasets/my_dataset/"]
  evaluation_method: "box"
  system_prompt:
    zh: |
      使用者會提供題目與選項 A、B、C、D，請以 \box{選項} 的格式回覆正確答案.

The ConfigurationManager in twinkle_eval/config.py processes this file, instantiates the LLM via LLMFactory.create_llm, and attaches the evaluation strategy.

Running the Evaluation Pipeline

Import TwinkleEvalRunner from the package root and execute the full workflow:

from twinkle_eval import TwinkleEvalRunner

# 1. Instantiate with config path

runner = TwinkleEvalRunner("config.yaml")

# 2. Load configuration and create results directory

runner.initialize()

# 3. Execute evaluation and export results

result_path = runner.run_evaluation(export_formats=["json", "html"])

print(f"Evaluation complete. Summary saved to: {result_path}")

The initialize() method must be called before run_evaluation() because it creates timestamps and prepares the output directory structure. The run_evaluation() method returns the path to the primary exported file, corresponding to the first entry produced by ResultsExporterFactory.export_results.

Advanced Usage Patterns

For scenarios requiring granular control, you can customize export formats or evaluate individual files without the full runner pipeline.

Customizing Export Formats

Pass a list of desired formats to run_evaluation():

# Export to CSV and Google Sheets (requires Google credentials in config)

result_files = runner.run_evaluation(export_formats=["csv", "google_sheets"])
print("Exported files:", result_files)

This programmatic approach mirrors the CLI's --export flag, allowing automated pipelines to specify output requirements dynamically.

Evaluating Single Files Manually

To evaluate a specific dataset file without the batch processing loop, use the Evaluator class directly:

from twinkle_eval.evaluators import Evaluator
from twinkle_eval.models import LLMFactory
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory
from twinkle_eval.config import load_config

# Load configuration and instantiate components

cfg = load_config("config.yaml")
llm = LLMFactory.create_llm(cfg["llm_api"]["type"], cfg)
strategy = EvaluationStrategyFactory.create_strategy(
    cfg["evaluation"]["evaluation_method"], 
    cfg
)

# Create evaluator with rate limiting

evaluator = Evaluator(llm, strategy, cfg)

# Process a single file

file_path, accuracy, results_file = evaluator.evaluate_file(
    "datasets/my_dataset/sample.jsonl",
    timestamp="20240101_1200",
    prompt_lang="zh"
)

print(f"Accuracy: {accuracy:.2%} | Results: {results_file}")

The Evaluator.evaluate_file method handles parallel request throttling via the internal RateLimiter class and returns the computed accuracy along with the path to the per-file JSONL results.

Accessing Results In-Memory

If you need to analyze results programmatically without reading from disk, capture the summary dictionary by loading the generated JSON file:

import json
import pathlib

summary_path = pathlib.Path("results") / f"results_{runner.start_time}.json"
with open(summary_path, "r", encoding="utf-8") as f:
    summary = json.load(f)

print(f"Average accuracy across all datasets: {summary['average_accuracy']}")

Google Workspace Integration

When your config.yaml includes a google_services section with valid Drive and Sheets credentials, TwinkleEvalRunner automatically uploads result files to Google Drive and publishes summary sheets. This requires no additional Python code beyond the standard run_evaluation() call, provided the service account credentials are accessible as specified in the repository documentation.

Summary

• Import TwinkleEvalRunner from twinkle_eval as the primary entry point for programmatic usage.
• Call initialize() before run_evaluation() to load configurations and create output directories.
• Use run_evaluation(export_formats=[...]) to specify output types including JSON, CSV, HTML, and Google Sheets.
• Access underlying classes (Evaluator, LLMFactory, EvaluationStrategyFactory) in twinkle_eval/evaluators.py and related modules for custom evaluation logic.
• Configure rate limiting and parallel processing through the YAML file consumed by ConfigurationManager in twinkle_eval/config.py.

Frequently Asked Questions

How do I install Twinkle Eval for programmatic use?

Install the package using pip install twinkle-eval. This provides access to TwinkleEvalRunner and all supporting factory classes defined in the ai-twinkle/eval repository.

What is the difference between initialize() and run_evaluation()?

The initialize() method in TwinkleEvalRunner loads the YAML configuration, instantiates the LLM and evaluation strategy via ConfigurationManager, and prepares the results directory. The run_evaluation() method executes the actual dataset processing, invokes the Evaluator class for parallel inference, and triggers result exports via ResultsExporterFactory.

Can I evaluate a single dataset file without using the full runner?

Yes. Import Evaluator from twinkle_eval/evaluators.py, instantiate it with an LLM from LLMFactory and a strategy from EvaluationStrategyFactory, then call evaluator.evaluate_file() with the specific file path. This bypasses the batch processing logic in TwinkleEvalRunner.

How does Twinkle Eval handle API rate limiting when used programmatically?

The Evaluator class automatically manages request throttling through its internal RateLimiter, which reads the api_rate_limit value from your configuration. This ensures compliance with provider constraints during parallel evaluation without requiring manual intervention in your Python code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}