# What Is an Experiment in Maka's Eval Semantics?

> Discover what an Experiment is in Apache Maka's eval semantics. Learn how this reproducible unit of work is defined and executed by the evaluation framework.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-12

---

**In Apache Maka's evaluation framework, an Experiment is the fundamental, reproducible unit of work defined by an `ExperimentSpec` JSON schema that the engine expands into concrete `ExperimentCell` objects—each representing a single subject-agent executing a single task for a specific repetition—before orchestrating execution through a pluggable `ExperimentExecutor` interface.**

An **Experiment** in Maka is not merely a configuration file but a complete declarative specification that drives the entire evaluation lifecycle. Located in the `apache/maka` repository within [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), the experiment abstraction encapsulates benchmarks, subjects, tasks, and execution constraints, enabling systematic agent evaluation through a structured expansion and execution pipeline.

## The ExperimentSpec: Declarative Structure

At the core of Maka's eval semantics lies the **`ExperimentSpec`** interface, a JSON object that fully describes the evaluation scenario. According to the source code in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), a valid spec must declare `schemaVersion: "maka.eval.v1"` and include the following components:

- **`id`** – A globally-unique identifier for the experiment run.
- **`benchmark`** – Defines the target benchmark with `id`, `version`, and a free-form `config` object.
- **`executor`** – Specifies *how* work is performed through a `kind` string and `config` parameters.
- **`execution`** – Runtime controls such as `maxConcurrentTaskGroups` for throttling parallel execution.
- **`subjects`** – The agents under evaluation. Each subject requires an `id`, a `kind` (either `"maka"` or `"external"`), a list of required `credentials`, and a `config` object.
- **`tasks`** – Individual work units defined by an `id`, textual `input`, and task-specific `config`.
- **`repetitions`** – An integer specifying how many times each subject-task pair should be executed.
- **`budget`** and **`verifier`** – Optional JSON objects for enforcing resource limits and validation rules.

## From Specification to Cells: The Expansion Logic

The transition from declaration to execution happens through the **`expandExperiment`** function (lines 84‑100 in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts)). This function performs a Cartesian product expansion:

```typescript
export function expandExperiment(spec: ExperimentSpec): ExperimentCell[]

```

The expansion follows this mapping:

1. **Tasks** × **Repetitions** × **Subjects** → **ExperimentCell[]**

Each resulting **`ExperimentCell`** represents a concrete, runnable instance. Cells carry a unique identifier constructed as `` `${task.id}::${repetition}::${subject.id}` ``, along with copies of the benchmark, executor, budget, and verifier from the original spec. This transformation ensures that every possible combination of subject, task, and repetition becomes an independent unit of work that the engine can schedule, retry, or distribute.

## Execution Flow and the ExperimentExecutor

Once expanded, the evaluation engine processes cells through a pipeline defined in [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts). The **`runExperiment`** function orchestrates the workflow:

1. **Parsing** – `parseExperimentSpec` (in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts)) validates the raw JSON against the schema.
2. **Workspace preparation** – `openExperimentDirectory` (in [`packages/eval/src/experiment-directory.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment-directory.ts)) initializes an on-disk workspace for artifacts and metadata.
3. **Cell grouping** – The engine groups cells according to `execution.maxConcurrentTaskGroups` to control parallelism.
4. **Execution** – For each cell, the engine invokes a user-supplied **`ExperimentExecutor`** implementing the interface from [`runner.ts`](https://github.com/apache/maka/blob/main/runner.ts). The executor implements `execute(cell)` and can optionally provide `validate()`, `prepare()`, and `cleanup()` hooks.

The **ExperimentExecutor** abstraction allows the same `ExperimentSpec` to run against different backends (local processes, Docker containers, cloud services) simply by swapping the `executor.kind` and corresponding implementation.

## Running Experiments: CLI and Programmatic APIs

Maka supports both command-line and programmatic execution. The CLI entry point in [`packages/eval/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/cli.ts) exposes the `run` command:

```bash
maka eval run --spec path/to/experiment.json --out-dir ./results

```

To execute programmatically, load and run the spec using the evaluation library:

```typescript
import { readFile } from 'fs/promises';
import { parseExperimentSpec } from './spec.js';
import { openExperimentDirectory } from './experiment-directory.js';
import { runExperiment, type ExperimentExecutor } from './runner.js';
import { resolve } from 'path';

// 1. Load and parse the spec
const raw = JSON.parse(await readFile('experiment.json', 'utf8'));
const spec = parseExperimentSpec(raw);

// 2. Prepare workspace
const workDir = await openExperimentDirectory(resolve('output'), spec);

// 3. Define an executor
const myExecutor: ExperimentExecutor = {
  async execute({ cell }) {
    console.log(`Running ${cell.subject.id} on ${cell.task.id}`);
    return { result: 'success' };
  }
};

// 4. Run
await runExperiment({ spec, executor: myExecutor });

```

A minimal `ExperimentSpec` requires only essential fields:

```json
{
  "schemaVersion": "maka.eval.v1",
  "id": "demo-exp-001",
  "benchmark": { "id": "bench-1", "version": "1.0", "config": {} },
  "executor": { "kind": "local", "config": {} },
  "execution": { "maxConcurrentTaskGroups": 4 },
  "subjects": [
    { "id": "agent-a", "kind": "maka", "credentials": [], "config": {} }
  ],
  "tasks": [
    { "id": "task-1", "input": "Evaluate this prompt", "config": {} }
  ],
  "repetitions": 3
}

```

## Summary

- An **Experiment** in Maka is defined by an **`ExperimentSpec`** JSON object that declaratively describes benchmarks, subjects, tasks, and execution constraints.
- The **`expandExperiment`** function (in [`experiment.ts`](https://github.com/apache/maka/blob/main/experiment.ts)) transforms the spec into an array of **`ExperimentCell`** objects through a Cartesian product of tasks, repetitions, and subjects.
- Each **ExperimentCell** is a concrete, runnable unit with a unique ID (`task::repetition::subject`) that carries its own copy of configuration contexts.
- The **`runExperiment`** function (in [`runner.ts`](https://github.com/apache/maka/blob/main/runner.ts)) drives execution by parsing the spec, preparing a workspace, and invoking a pluggable **`ExperimentExecutor`** for each cell.
- The **`maxConcurrentTaskGroups`** parameter controls parallelism by grouping cells into execution waves.
- Execution can be triggered via the **CLI** ([`cli.ts`](https://github.com/apache/maka/blob/main/cli.ts)) or programmatically using the TypeScript API.

## Frequently Asked Questions

### What is the difference between ExperimentSpec and ExperimentCell?

An **`ExperimentSpec`** is the high-level, user-authored JSON configuration that describes the entire evaluation scenario, including all subjects, tasks, and parameters. An **`ExperimentCell`**, defined in the same [`experiment.ts`](https://github.com/apache/maka/blob/main/experiment.ts) file, is a single concrete instance generated by `expandExperiment` that represents one specific subject executing one specific task during one specific repetition. While the spec is declarative and comprehensive, cells are the atomic units processed by the execution engine.

### How does Maka handle concurrent execution of experiments?

The **`execution.maxConcurrentTaskGroups`** field in the `ExperimentSpec` dictates concurrency. During the expansion phase in [`runner.ts`](https://github.com/apache/maka/blob/main/runner.ts), the engine groups `ExperimentCell` objects into batches sized according to this limit. The `ExperimentExecutor` then processes cells within a group concurrently, while the engine manages backpressure and resource limits across groups.

### What is the role of the ExperimentExecutor interface?

The **`ExperimentExecutor`** interface (defined in [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts)) abstracts the actual execution logic from the experiment definition. Implementations must provide an `execute` method that receives an `ExperimentCell` and returns results, but can also implement optional lifecycle hooks: `validate()` for pre-flight checks, `prepare()` for setup, and `cleanup()` for resource disposal. This allows the same experiment specification to run on local machines, containerized environments, or remote cloud services by swapping executor implementations.

### How does the expandExperiment function generate cell IDs?

According to lines 84‑100 of [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), the `expandExperiment` function constructs each cell's unique identifier using the template string `` `${task.id}::${repetition}::${subject.id}` ``. This guarantees uniqueness across the Cartesian product of all tasks, all repetition indices (from 0 to `repetitions-1`), and all subjects, ensuring the execution engine can track, log, and retry individual runs without ambiguity.