# Internal Architecture of the Evaluator Class and Multi-File Processing in Twinkle Eval

> Explore the internal architecture of the Evaluator class and discover how Twinkle Eval handles multi-file processing for efficient LLM evaluations with parallel calls and API throttling.

- Repository: [Twinkle AI/eval](https://github.com/ai-twinkle/eval)
- Tags: internals
- Published: 2026-02-23

---

**The `Evaluator` class in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py) orchestrates single-file evaluations using a `ThreadPoolExecutor` for parallel LLM calls, a `RateLimiter` for API throttling, and an `EvaluationStrategy` for answer extraction, while `TwinkleEvalRunner` in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) coordinates sequential multi-file processing with statistical aggregation.**

The Twinkle Eval framework provides a robust pipeline for benchmarking large language models against standardized datasets. At the heart of this system lies the `Evaluator` class, which manages the complete lifecycle of a single evaluation run—from prompt construction to accuracy calculation. Understanding the internal architecture of the Evaluator class reveals how the system efficiently processes multiple dataset files while respecting API rate limits and maintaining detailed audit trails.

## Core Components of the Evaluator Class

### Initialization and Rate Limiting

The `Evaluator.__init__` method (lines 31-36 in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py)) receives three critical dependencies: an LLM object, an `EvaluationStrategy` instance, and the global configuration dictionary. During construction, it instantiates a `RateLimiter` object (lines 15-20) that enforces the per-second API quota defined in `config["llm_api"]["api_rate_limit"]`. This ensures that parallel execution threads never exceed the provider's request thresholds.

### Optional Option Shuffling

When `config["evaluation"]["shuffle_options"]` is enabled, the evaluator invokes `shuffle_question_options` (lines 38-61) to randomly reorder answer choices while preserving the mapping to the correct answer. This feature prevents positional bias in multiple-choice evaluations by ensuring the LLM cannot rely on option ordering patterns.

## Single-File Evaluation Pipeline

### Dataset Loading and Preparation

The `evaluate_file` method (lines 63-147) serves as the primary entry point for processing individual dataset files. It begins by loading the file via the `Dataset` helper class (`Dataset(file_path)` at line 64), which provides an iterator over JSON-Lines formatted questions. For each question, the method constructs a prompt string (`question_text`) concatenating the question body with all available answer options (lines 79-85).

### Parallel LLM Execution with ThreadPoolExecutor

To maximize throughput, the evaluator creates a `ThreadPoolExecutor` context manager (line 71) that submits concurrent LLM calls for each question. Before every submission, the code invokes `RateLimiter.wait()` (line 93) to block until the API quota permits another request. The actual LLM invocation occurs via `executor.submit(self.llm.call, question_text, prompt_lang)` (line 94), with futures stored in a `future_to_data` mapping (line 96) to preserve question metadata.

### Answer Extraction and Scoring

As futures complete (`as_completed(future_tasks)`, lines 98-100), the evaluator extracts the raw LLM response and delegates answer parsing to the configured strategy via `self.evaluation_strategy.extract_answer(content)` (line 109). The predicted answer is compared against the ground-truth (`correct_answer`) to update a running tally of correct responses (lines 111-118). Detailed per-question metadata—including the full prompt, raw output, token usage statistics, and correctness boolean—is appended to `detailed_results` (lines 120-133).

### Result Persistence

Upon completion of all futures, the method calculates overall accuracy as `total_correct / total_questions` (line 135) and serializes the `detailed_results` list to a JSON-Lines file under the `results/` directory (lines 137-146). The method returns a tuple containing the original file path, accuracy score, and result file path.

## Orchestrating Multiple Dataset Files

### File Discovery and Batch Processing

While the `Evaluator` handles single files, `TwinkleEvalRunner._evaluate_dataset` in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) (lines 84-124) manages the broader evaluation campaign. It first discovers all evaluation files via `find_all_evaluation_files(dataset_path)` (line 101), then iterates through each file path calling `evaluator.evaluate_file(...)` within a try/except block (line 115).

### Repeated Runs and Statistical Aggregation

The runner supports statistical robustness through the `repeat_runs` configuration parameter (line 102), executing multiple evaluation passes per file. After collecting accuracy scores across all repetitions (lines 117-119), it calculates mean accuracy and standard deviation (lines 124-128) for each file. A progress indicator displays completion percentage (lines 141-144), and final dataset-wide aggregates are computed (lines 149-155) before returning comprehensive metrics.

## Practical Implementation Examples

```python

# Create an evaluator for a single LLM and strategy

from twinkle_eval.evaluators import Evaluator
from twinkle_eval.models import LLMFactory
from twinkle_eval.evaluation_strategies import EvaluationStrategyFactory

llm = LLMFactory.create("openai", api_key="…", base_url="https://api.openai.com")
strategy = EvaluationStrategyFactory.create("pattern_matching", {})
config = {
    "llm_api": {"api_rate_limit": 5},
    "evaluation": {"shuffle_options": True},
}
evaluator = Evaluator(llm, strategy, config)

# Evaluate a single dataset file

file_path = "datasets/mmlu/abstract_algebra.jsonl"
timestamp = "20240223_001"
accuracy, result_file = evaluator.evaluate_file(file_path, timestamp)[1:]
print(f"File accuracy: {accuracy:.2%}, results saved to {result_file}")

```

```python

# Run the full evaluation across all datasets defined in config.yaml

from twinkle_eval.main import TwinkleEvalRunner

runner = TwinkleEvalRunner(config_path="config.yaml")
runner.initialize()

# Export both JSON and HTML reports

runner.run_evaluation(export_formats=["json", "html"])

```

## Summary

- The `Evaluator` class encapsulates single-file evaluation logic in [`twinkle_eval/evaluators.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/evaluators.py), utilizing `ThreadPoolExecutor` for parallel processing and `RateLimiter` for API compliance.
- The `evaluate_file` method handles the complete pipeline: dataset loading, optional option shuffling, parallel LLM inference, answer extraction via `EvaluationStrategy`, and JSON-Lines result serialization.
- `TwinkleEvalRunner` in [`twinkle_eval/main.py`](https://github.com/ai-twinkle/eval/blob/main/twinkle_eval/main.py) coordinates multi-file campaigns through `_evaluate_dataset`, supporting repeated runs and statistical aggregation across dataset directories.
- The architecture separates concerns between per-file execution (parallelism, rate limiting) and cross-file orchestration (batching, progress tracking, metrics aggregation).

## Frequently Asked Questions

### How does the Evaluator class handle API rate limits when processing questions in parallel?

The `Evaluator` instantiates a `RateLimiter` during initialization that tracks the `api_rate_limit` value from the configuration. Before submitting each LLM call to the `ThreadPoolExecutor`, the code explicitly calls `RateLimiter.wait()` to block execution until the per-second quota allows another request, ensuring parallel threads never exceed API thresholds.

### What is the purpose of the shuffle_options feature in the Evaluator?

The `shuffle_question_options` function (lines 38-61) randomly reorders multiple-choice answer options while maintaining the correct answer mapping. This prevents the LLM from exploiting positional biases or patterns in option ordering that might artificially inflate benchmark scores.

### How does TwinkleEvalRunner process multiple dataset files sequentially?

`TwinkleEvalRunner._evaluate_dataset` discovers all files in a dataset directory using `find_all_evaluation_files`, then loops through each path calling `evaluator.evaluate_file()` individually. This sequential file processing combined with internal parallel question evaluation provides controlled resource utilization while supporting statistical aggregation across files and repeated runs.

### Where are the detailed evaluation results stored after processing?

The `evaluate_file` method writes comprehensive per-question results—including prompts, responses, usage statistics, and correctness flags—to JSON-Lines files in the `results/` directory (lines 137-146). The file path is returned to the caller and tracked by `TwinkleEvalRunner` for final report generation.