# How Apache Maka Manages Experiments and Evaluations: A Technical Deep Dive

> Learn how Apache Maka manages experiments and evaluations using JSON specs and a dedicated framework. Explore schema validation, concurrency, and distributed results aggregation for efficient ML workflows.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-08

---

**Apache Maka treats experiments and evaluations as first-class assets described by JSON specifications and executed through a dedicated evaluation framework in `packages/eval` that validates schemas, manages concurrency, and aggregates results across distributed runtime hosts.**

Apache Maka provides a robust framework for managing experiments and evaluations through a declarative, version-controlled approach. The system treats each experiment as a structured asset defined by JSON specifications and orchestrated by a dedicated evaluation engine located in the `packages/eval` directory. This architecture ensures reproducible, isolated test execution across diverse runtime environments while maintaining strict consistency between on-disk configurations and in-memory representations.

## Experiment Definition and Schema Validation

Every experiment in Maka lives under the top-level `experiments/` folder and is defined by a specification file named [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json). This JSON file declares the **experiment ID**, a list of **subjects** (the services or modules under test), and the **tasks** (actions each subject should perform).

The schema is enforced by [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), which validates the `schemaVersion` field (expected value: `maka.eval.v1`) and guarantees that every experiment contains at least one subject and one task. This validation ensures structural integrity before any execution begins.

```typescript
// Example experiment.json structure validated by the framework
{
  "schemaVersion": "maka.eval.v1",
  "id": "terminal-bench-2.1-deepseek-v4-flash-eight-arm",
  "subjects": ["service-a", "service-b"],
  "tasks": ["benchmark", "validate"]
}

```

## Directory Handling and Spec Consistency

Before execution, the framework ensures the on-disk specification matches the expected in-memory representation. The [`packages/eval/src/experiment-directory.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment-directory.ts) module opens the experiment directory, reads the [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) file, and performs strict consistency checks.

If the specification differs from the expected structure or version, the framework throws an error immediately. This prevents silent configuration drift and ensures that experiments remain reproducible across different execution environments.

## The Evaluation Execution Pipeline

Running an experiment involves two core components that transform specifications into executable units. First, [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts) parses the specification and builds a **cell** for each `(subject, task)` pair, creating a matrix of individual test units.

These cells are then handed to the **harness executor** ([`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts)), which delegates execution to the appropriate runtime environment. Supported runtimes include Docker containers, Windows sandboxes, and remote hosts, allowing flexible deployment across heterogeneous infrastructure.

```typescript
// Loading and running an experiment using the Maka evaluation API
import { openExperimentDirectory, runExperiment } from '@maka/eval';

// 1. Load the experiment directory
const expDir = await openExperimentDirectory(
  new URL('../experiments/terminal-bench-2.1-deepseek-v4-flash-eight-arm.json', import.meta.url)
);

// 2. Execute the experiment
const result = await runExperiment(expDir);

// 3. Access results
console.log(`Experiment ${result.experimentId} completed ${result.cells} cells`);

```

## Result Collection and CLI Reporting

Once execution completes, [`packages/eval/src/result.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/result.ts) aggregates outcomes into a structured result object. This object contains the `experimentId`, the total count of completed cells, and a count of any **incomplete** cells that failed or timed out.

The command-line interface ([`packages/eval/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/cli.ts)) consumes these results and prints a concise JSON line summarizing the run, following this format:

```json
{ "experimentId": "terminal-bench-2.1", "cells": 12, "incomplete": 0 }

```

This machine-readable output enables easy integration with CI/CD pipelines and downstream analysis tools.

## Concurrency Control and State Safety

To prevent data corruption from concurrent executions, Maka implements exclusive write access through [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts). Before starting an experiment, the framework writes a temporary "attempt store" to disk.

This mechanism ensures that only one writer can be active for a given experiment directory at any time. If a second process attempts to run the same experiment while it is already executing, the attempt store blocks the new execution, preventing race conditions and state corruption.

## Runtime Integration and Experimental Features

The evaluation harness communicates with various runtime hosts (such as [`packages/runtime-host/src/protocol/web-search.ts`](https://github.com/apache/maka/blob/main/packages/runtime-host/src/protocol/web-search.ts)) through well-defined contracts. The framework supports toggling experimental features without code changes by using headers like `experimental_disabled` or `OpenAI-Beta`, allowing gradual rollout of new capabilities while maintaining stability in production evaluations.

## Summary

- **Experiments and evaluations** in Maka are defined by JSON specifications ([`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json)) stored in the `experiments/` directory and validated against the `maka.eval.v1` schema.
- The framework enforces strict consistency between on-disk specifications and in-memory representations through [`packages/eval/src/experiment-directory.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment-directory.ts).
- Execution is orchestrated by [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts) building cells and [`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts) delegating to runtimes like Docker or Windows sandboxes.
- Results are aggregated in [`packages/eval/src/result.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/result.ts) and emitted as JSON by the CLI for pipeline integration.
- Concurrency protection via [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts) prevents simultaneous writes and data corruption.
- Runtime hosts support feature flags through HTTP headers, enabling safe testing of experimental capabilities.

## Frequently Asked Questions

### What is the structure of a Maka experiment specification?

A Maka experiment specification is a JSON file named [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) that declares a `schemaVersion` (set to `maka.eval.v1`), a unique experiment `id`, an array of **subjects** representing the code under test, and an array of **tasks** defining the operations to perform. The [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts) module validates this structure, ensuring at least one subject and one task exist before execution proceeds.

### How does Maka prevent concurrent experiment runs?

Maka prevents concurrent runs through the **attempt store** mechanism implemented in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts). Before execution begins, the framework writes a temporary attempt store to the experiment directory. This acts as a lock file, ensuring only one process can write to the directory at a time. Subsequent attempts to run the same experiment will fail fast rather than corrupt shared state.

### What runtimes does Maka support for evaluations?

According to the source code in [`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts), Maka supports multiple runtime environments including **Docker containers**, **Windows sandboxes**, and **remote hosts**. The harness executor abstracts these environments, allowing the same experiment specification to run across different isolation mechanisms without modification.

### How are experimental features handled in the evaluation framework?

Experimental features are managed through HTTP headers such as `experimental_disabled` or `OpenAI-Beta` in runtime host protocols (as seen in [`packages/runtime-host/src/protocol/web-search.ts`](https://github.com/apache/maka/blob/main/packages/runtime-host/src/protocol/web-search.ts)). This header-based approach allows operators to toggle experimental capabilities on or off without deploying new code, providing a safe mechanism for testing new evaluation methods while maintaining production stability.