# How to Use Twinkle Eval Programmatically with the Python API: A Complete Guide

> Learn to use Twinkle Eval programmatically with Python. This guide shows you how to import TwinkleEvalRunner, initialize, and run evaluations for your datasets.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: how-to-guide
- Published: 2026-02-23

---

**To use Twinkle Eval programmatically, import `TwinkleEvalRunner` from the `twinkle-eval` package, instantiate it with a configuration file path, call `initialize()` to load settings, and execute `run_evaluation()` to process datasets and export results.**

The `ai-twinkle/eval` repository provides a modular framework for evaluating large-language-model (LLM) responses against standardized datasets. While the CLI offers quick access, the Python API gives you fine-grained control over configuration loading, LLM instantiation, evaluation strategies, and result export formats. This guide demonstrates how to use Twinkle Eval programmatically using the core classes defined in the source code.

## Installation and Prerequisites

Install the package from PyPI to access the programmatic interface:

```bash
pip install twinkle-eval

```

The [`setup.py`](https://github.com/ai-twinkle/eval/blob/main/setup.py) in the repository publishes this package name, ensuring you receive the correct distribution with all dependencies.

## Core Architecture Overview

Understanding the key classes helps you use Twinkle Eval programmatically with precision. The framework follows a factory pattern for extensibility:

- **`TwinkleEvalRunner`** ([`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py), lines 61-84): The primary entry point that orchestrates the entire pipeline.
- **`ConfigurationManager`** / `load_config` ([`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py)): Parses YAML configuration and validates required sections.
- **`LLMFactory`** ([`twinkle_eval/models.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/models.py), lines 9-33): Creates LLM instances (OpenAI-compatible) with validated API credentials.
- **`EvaluationStrategyFactory`** ([`twinkle_eval/evaluation_strategies.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluation_strategies.py)): Returns concrete strategies like `BoxExtractionStrategy` or `PatternMatchingStrategy`.
- **`Evaluator`** ([`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py), lines 31-48): Handles parallel execution, rate limiting, and answer extraction.
- **`ResultsExporterFactory`** ([`twinkle_eval/results_exporters.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/results_exporters.py)): Converts summaries to JSON, CSV, HTML, or Google Sheets.

## Basic Programmatic Usage

The standard workflow involves three steps: instantiate the runner, initialize internal state, and execute the evaluation.

### Loading Configuration

First, create a YAML configuration file that specifies your LLM API details, model parameters, and evaluation method:

```yaml
llm_api:
  base_url: "http://localhost:8000/v1"
  api_key: "YOUR_API_KEY"
  api_rate_limit: 2
model:
  name: "gpt-4o-mini"
  temperature: 0.0
  top_p: 0.9
evaluation:
  dataset_paths: ["datasets/my_dataset/"]
  evaluation_method: "box"
  system_prompt:
    zh: |
      使用者會提供題目與選項 A、B、C、D，請以 \box{選項} 的格式回覆正確答案.

```

The `ConfigurationManager` in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py) processes this file, instantiates the LLM via `LLMFactory.create_llm`, and attaches the evaluation strategy.

### Running the Evaluation Pipeline

Import `TwinkleEvalRunner` from the package root and execute the full workflow:

```python
from twinkle_eval import TwinkleEvalRunner

# 1. Instantiate with config path

runner = TwinkleEvalRunner("config.yaml")

# 2. Load configuration and create results directory

runner.initialize()

# 3. Execute evaluation and export results

result_path = runner.run_evaluation(export_formats=["json", "html"])

print(f"Evaluation complete. Summary saved to: {result_path}")

```

The `initialize()` method must be called before `run_evaluation()` because it creates timestamps and prepares the output directory structure. The `run_evaluation()` method returns the path to the primary exported file, corresponding to the first entry produced by `ResultsExporterFactory.export_results`.

## Advanced Usage Patterns

For scenarios requiring granular control, you can customize export formats or evaluate individual files without the full runner pipeline.

### Customizing Export Formats

Pass a list of desired formats to `run_evaluation()`:

```python

# Export to CSV and Google Sheets (requires Google credentials in config)

result_files = runner.run_evaluation(export_formats=["csv", "google_sheets"])
print("Exported files:", result_files)

```

This programmatic approach mirrors the CLI's `--export` flag, allowing automated pipelines to specify output requirements dynamically.

### Evaluating Single Files Manually

To evaluate a specific dataset file without the batch processing loop, use the `Evaluator` class directly:

```python
from twinkle_eval.evaluators import Evaluator
from twinkle_eval.models import LLMFactory
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory
from twinkle_eval.config import load_config

# Load configuration and instantiate components

cfg = load_config("config.yaml")
llm = LLMFactory.create_llm(cfg["llm_api"]["type"], cfg)
strategy = EvaluationStrategyFactory.create_strategy(
    cfg["evaluation"]["evaluation_method"], 
    cfg
)

# Create evaluator with rate limiting

evaluator = Evaluator(llm, strategy, cfg)

# Process a single file

file_path, accuracy, results_file = evaluator.evaluate_file(
    "datasets/my_dataset/sample.jsonl",
    timestamp="20240101_1200",
    prompt_lang="zh"
)

print(f"Accuracy: {accuracy:.2%} | Results: {results_file}")

```

The `Evaluator.evaluate_file` method handles parallel request throttling via the internal `RateLimiter` class and returns the computed accuracy along with the path to the per-file JSONL results.

### Accessing Results In-Memory

If you need to analyze results programmatically without reading from disk, capture the summary dictionary by loading the generated JSON file:

```python
import json
import pathlib

summary_path = pathlib.Path("results") / f"results_{runner.start_time}.json"
with open(summary_path, "r", encoding="utf-8") as f:
    summary = json.load(f)

print(f"Average accuracy across all datasets: {summary['average_accuracy']}")

```

## Google Workspace Integration

When your [`config.yaml`](https://github.com/ai-twinkle/eval/blob/main/config.yaml) includes a `google_services` section with valid Drive and Sheets credentials, `TwinkleEvalRunner` automatically uploads result files to Google Drive and publishes summary sheets. This requires no additional Python code beyond the standard `run_evaluation()` call, provided the service account credentials are accessible as specified in the repository documentation.

## Summary

- **Import `TwinkleEvalRunner`** from `twinkle_eval` as the primary entry point for programmatic usage.
- **Call `initialize()`** before `run_evaluation()` to load configurations and create output directories.
- **Use `run_evaluation(export_formats=[...])`** to specify output types including JSON, CSV, HTML, and Google Sheets.
- **Access underlying classes** (`Evaluator`, `LLMFactory`, `EvaluationStrategyFactory`) in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) and related modules for custom evaluation logic.
- **Configure rate limiting and parallel processing** through the YAML file consumed by `ConfigurationManager` in [`twinkle_eval/config.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/config.py).

## Frequently Asked Questions

### How do I install Twinkle Eval for programmatic use?

Install the package using `pip install twinkle-eval`. This provides access to `TwinkleEvalRunner` and all supporting factory classes defined in the `ai-twinkle/eval` repository.

### What is the difference between `initialize()` and `run_evaluation()`?

The `initialize()` method in `TwinkleEvalRunner` loads the YAML configuration, instantiates the LLM and evaluation strategy via `ConfigurationManager`, and prepares the results directory. The `run_evaluation()` method executes the actual dataset processing, invokes the `Evaluator` class for parallel inference, and triggers result exports via `ResultsExporterFactory`.

### Can I evaluate a single dataset file without using the full runner?

Yes. Import `Evaluator` from [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py), instantiate it with an LLM from `LLMFactory` and a strategy from `EvaluationStrategyFactory`, then call `evaluator.evaluate_file()` with the specific file path. This bypasses the batch processing logic in `TwinkleEvalRunner`.

### How does Twinkle Eval handle API rate limiting when used programmatically?

The `Evaluator` class automatically manages request throttling through its internal `RateLimiter`, which reads the `api_rate_limit` value from your configuration. This ensures compliance with provider constraints during parallel evaluation without requiring manual intervention in your Python code.