What Is an Experiment in Maka's Eval Semantics?

In Apache Maka's evaluation framework, an Experiment is the fundamental, reproducible unit of work defined by an ExperimentSpec JSON schema that the engine expands into concrete ExperimentCell objects—each representing a single subject-agent executing a single task for a specific repetition—before orchestrating execution through a pluggable ExperimentExecutor interface.

An Experiment in Maka is not merely a configuration file but a complete declarative specification that drives the entire evaluation lifecycle. Located in the apache/maka repository within packages/eval/src/experiment.ts, the experiment abstraction encapsulates benchmarks, subjects, tasks, and execution constraints, enabling systematic agent evaluation through a structured expansion and execution pipeline.

The ExperimentSpec: Declarative Structure

At the core of Maka's eval semantics lies the ExperimentSpec interface, a JSON object that fully describes the evaluation scenario. According to the source code in packages/eval/src/experiment.ts, a valid spec must declare schemaVersion: "maka.eval.v1" and include the following components:

• id – A globally-unique identifier for the experiment run.
• benchmark – Defines the target benchmark with id, version, and a free-form config object.
• executor – Specifies how work is performed through a kind string and config parameters.
• execution – Runtime controls such as maxConcurrentTaskGroups for throttling parallel execution.
• subjects – The agents under evaluation. Each subject requires an id, a kind (either "maka" or "external"), a list of required credentials, and a config object.
• tasks – Individual work units defined by an id, textual input, and task-specific config.
• repetitions – An integer specifying how many times each subject-task pair should be executed.
• budget and verifier – Optional JSON objects for enforcing resource limits and validation rules.

From Specification to Cells: The Expansion Logic

The transition from declaration to execution happens through the expandExperiment function (lines 84‑100 in packages/eval/src/experiment.ts). This function performs a Cartesian product expansion:

export function expandExperiment(spec: ExperimentSpec): ExperimentCell[]

The expansion follows this mapping:

1. Tasks × Repetitions × Subjects → ExperimentCell[]

Each resulting ExperimentCell represents a concrete, runnable instance. Cells carry a unique identifier constructed as `${task.id}::${repetition}::${subject.id}`, along with copies of the benchmark, executor, budget, and verifier from the original spec. This transformation ensures that every possible combination of subject, task, and repetition becomes an independent unit of work that the engine can schedule, retry, or distribute.

Execution Flow and the ExperimentExecutor

Once expanded, the evaluation engine processes cells through a pipeline defined in packages/eval/src/runner.ts. The runExperiment function orchestrates the workflow:

1. Parsing – parseExperimentSpec (in packages/eval/src/spec.ts) validates the raw JSON against the schema.
2. Workspace preparation – openExperimentDirectory (in packages/eval/src/experiment-directory.ts) initializes an on-disk workspace for artifacts and metadata.
3. Cell grouping – The engine groups cells according to execution.maxConcurrentTaskGroups to control parallelism.
4. Execution – For each cell, the engine invokes a user-supplied ExperimentExecutor implementing the interface from runner.ts. The executor implements execute(cell) and can optionally provide validate(), prepare(), and cleanup() hooks.

The ExperimentExecutor abstraction allows the same ExperimentSpec to run against different backends (local processes, Docker containers, cloud services) simply by swapping the executor.kind and corresponding implementation.

Running Experiments: CLI and Programmatic APIs

Maka supports both command-line and programmatic execution. The CLI entry point in packages/eval/src/cli.ts exposes the run command:

maka eval run --spec path/to/experiment.json --out-dir ./results

To execute programmatically, load and run the spec using the evaluation library:

import { readFile } from 'fs/promises';
import { parseExperimentSpec } from './spec.js';
import { openExperimentDirectory } from './experiment-directory.js';
import { runExperiment, type ExperimentExecutor } from './runner.js';
import { resolve } from 'path';

// 1. Load and parse the spec
const raw = JSON.parse(await readFile('experiment.json', 'utf8'));
const spec = parseExperimentSpec(raw);

// 2. Prepare workspace
const workDir = await openExperimentDirectory(resolve('output'), spec);

// 3. Define an executor
const myExecutor: ExperimentExecutor = {
  async execute({ cell }) {
    console.log(`Running ${cell.subject.id} on ${cell.task.id}`);
    return { result: 'success' };
  }
};

// 4. Run
await runExperiment({ spec, executor: myExecutor });

A minimal ExperimentSpec requires only essential fields:

{
  "schemaVersion": "maka.eval.v1",
  "id": "demo-exp-001",
  "benchmark": { "id": "bench-1", "version": "1.0", "config": {} },
  "executor": { "kind": "local", "config": {} },
  "execution": { "maxConcurrentTaskGroups": 4 },
  "subjects": [
    { "id": "agent-a", "kind": "maka", "credentials": [], "config": {} }
  ],
  "tasks": [
    { "id": "task-1", "input": "Evaluate this prompt", "config": {} }
  ],
  "repetitions": 3
}

Summary

• An Experiment in Maka is defined by an ExperimentSpec JSON object that declaratively describes benchmarks, subjects, tasks, and execution constraints.
• The expandExperiment function (in experiment.ts) transforms the spec into an array of ExperimentCell objects through a Cartesian product of tasks, repetitions, and subjects.
• Each ExperimentCell is a concrete, runnable unit with a unique ID (task::repetition::subject) that carries its own copy of configuration contexts.
• The runExperiment function (in runner.ts) drives execution by parsing the spec, preparing a workspace, and invoking a pluggable ExperimentExecutor for each cell.
• The maxConcurrentTaskGroups parameter controls parallelism by grouping cells into execution waves.
• Execution can be triggered via the CLI (cli.ts) or programmatically using the TypeScript API.

Frequently Asked Questions

What is the difference between ExperimentSpec and ExperimentCell?

An ExperimentSpec is the high-level, user-authored JSON configuration that describes the entire evaluation scenario, including all subjects, tasks, and parameters. An ExperimentCell, defined in the same experiment.ts file, is a single concrete instance generated by expandExperiment that represents one specific subject executing one specific task during one specific repetition. While the spec is declarative and comprehensive, cells are the atomic units processed by the execution engine.

How does Maka handle concurrent execution of experiments?

The execution.maxConcurrentTaskGroups field in the ExperimentSpec dictates concurrency. During the expansion phase in runner.ts, the engine groups ExperimentCell objects into batches sized according to this limit. The ExperimentExecutor then processes cells within a group concurrently, while the engine manages backpressure and resource limits across groups.

What is the role of the ExperimentExecutor interface?

The ExperimentExecutor interface (defined in packages/eval/src/runner.ts) abstracts the actual execution logic from the experiment definition. Implementations must provide an execute method that receives an ExperimentCell and returns results, but can also implement optional lifecycle hooks: validate() for pre-flight checks, prepare() for setup, and cleanup() for resource disposal. This allows the same experiment specification to run on local machines, containerized environments, or remote cloud services by swapping executor implementations.

How does the expandExperiment function generate cell IDs?

According to lines 84‑100 of packages/eval/src/experiment.ts, the expandExperiment function constructs each cell's unique identifier using the template string `${task.id}::${repetition}::${subject.id}`. This guarantees uniqueness across the Cartesian product of all tasks, all repetition indices (from 0 to repetitions-1), and all subjects, ensuring the execution engine can track, log, and retry individual runs without ambiguity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}