# How to Configure an Experiment in Maka’s Eval Framework: Complete Guide with Examples

> Configure Maka experiments easily using experiment.json. This guide shows how to define benchmarks, executors, subjects, and tasks to run your benchmarks.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: how-to-guide
- Published: 2026-09-09

---

**Configure a Maka experiment by writing an [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) spec that defines your benchmark, executor, subjects, and tasks, then execute it via `maka eval run experiment.json --out <dir>`.**

The Apache Maka repository provides a rigorous Eval framework for reproducible AI benchmarking. To configure an experiment in Maka's Eval Framework, you create a declarative JSON specification that the system validates against a hand-written decoder, freezes for immutability, and expands into a Cartesian product of execution cells. This guide walks through the spec structure, validation logic, and execution flow using actual source paths from the codebase.

## Understanding the Experiment Specification Format

The experiment specification is a JSON file that completely describes how to run a benchmark. According to the source code in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts), the framework uses a hand-written decoder—**not** an external JSON Schema library—to parse and validate the spec, ensuring strict type safety and explicit versioning.

### Required Schema Fields

Every [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) must include these top-level keys:

- **schemaVersion**: Must be the exact string `"maka.eval.v1"`. Unknown top-level keys cause immediate validation errors.
- **id**: A human-readable string identifier for the experiment.
- **benchmark**: An object containing `id`, `version`, and an opaque `config` object for benchmark-specific settings.
- **executor**: Specifies the execution backend with `kind` (e.g., `"harbor"` or `"pier"`) and `config`.
- **subjects**: An array of subject definitions, each with `id`, `kind` (`"maka"` or `"external"`), optional `credentials`, and `config`.
- **tasks**: An array of task objects, each requiring `id`, `input` (the prompt or payload), and `config`.
- **repetitions**: A positive integer specifying how many times to sample each cell.
- **budget**: Resource limits passed as a `JsonObject` to the executor.
- **verifier**: Benchmark-specific verification logic as a `JsonObject`.

### Optional Execution Parameters

The `execution` field accepts properties like `maxConcurrentTaskGroups` to limit parallelism. If omitted, the framework defaults to single-threaded execution.

## Step-by-Step Configuration Guide

### Step 1: Define the Benchmark and Executor

Start your [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) by specifying which benchmark to run and which executor will run it. In [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts), the decoder validates that the `benchmark.id` and `executor.kind` are present and correctly typed.

```json
{
  "schemaVersion": "maka.eval.v1",
  "id": "terminal-benchmark-v2",
  "benchmark": {
    "id": "terminal-bench",
    "version": "2.1",
    "config": {}
  },
  "executor": {
    "kind": "harbor",
    "config": {}
  }
}

```

### Step 2: Configure Subjects and Tasks

Subjects are the models or systems under test. Tasks define the inputs. The framework expands these into a Cartesian product in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), creating cells for every combination of `task × subject × repetition`.

```json
{
  "subjects": [
    {
      "id": "maka-subject",
      "kind": "maka",
      "credentials": [],
      "config": {}
    }
  ],
  "tasks": [
    {
      "id": "task-1",
      "input": "Write a short poem about clouds.",
      "config": {}
    }
  ]
}

```

### Step 3: Set Repetitions and Budget

Control statistical robustness and resource consumption:

```json
{
  "repetitions": 3,
  "budget": {
    "maxSteps": 5000
  },
  "verifier": {
    "type": "exact-match",
    "expected": ".*"
  }
}

```

The `repetitions` value directly determines how many samples the runner collects per cell before aggregating results.

## Running the Experiment

Once configured, execute via the CLI entry point defined in [`packages/cli/src/cli-core.ts`](https://github.com/apache/maka/blob/main/packages/cli/src/cli-core.ts):

```bash

# Create output directory

mkdir -p .maka-eval/run-001

# Run the experiment

maka eval run experiment.json --out .maka-eval/run-001

```

The CLI performs prerequisite checks—Docker images, relay files, environment variables—before starting any trials. If a prerequisite is missing, it aborts immediately.

### Rerunning Individual Cells

Use the `--cell <cell-id>` flag to replace a single failed cell without re-running the entire experiment:

```bash
maka eval run experiment.json --out .maka-eval/run-001 --cell "task-1::1::maka-subject"

```

## Extending the Framework with Custom Executors

For advanced use cases, implement the `ExperimentExecutor` or `SubjectAdapter` interfaces from [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts) and inject them via `EvalCliDependencies` in [`packages/eval/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/cli.ts).

```typescript
import { runMakaEvalCli } from '@maka/eval';
import { createMyExecutor } from './my-executor';

runMakaEvalCli({
  specPath: 'experiment.json',
  outPath: '.maka-eval/run-custom',
  loadExecutor: (spec, specPath) => createMyExecutor(spec, specPath),
  subjects: [],
  stdout: console.log,
  stderr: console.error,
});

```

This allows integration with proprietary schedulers or custom model endpoints while retaining the framework's validation, directory management in [`packages/eval/src/experiment-directory.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment-directory.ts), and result aggregation.

## Summary

- **Configure an experiment in Maka's Eval Framework** by creating an [`experiment.json`](https://github.com/apache/maka/blob/main/experiment.json) with `schemaVersion: "maka.eval.v1"` and defining benchmark, executor, subjects, tasks, repetitions, budget, and verifier.
- **Validation** occurs through a strict hand-written decoder in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) that rejects unknown keys.
- **Immutability** is enforced when the `ExperimentSpec` is frozen in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), ensuring reproducibility.
- **Execution** expands the spec into cells (task × repetition × subject) and persists results to an append-only log managed by [`packages/eval/src/experiment-directory.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment-directory.ts).
- **Customization** is supported via the `EvalCliDependencies` interface for injecting custom executors or subjects.

## Frequently Asked Questions

### What is the required schema version for Maka experiment specs?

The `schemaVersion` field must be exactly `"maka.eval.v1"`. The hand-written decoder in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) enforces this string literal and rejects unknown top-level keys to prevent configuration drift.

### How does Maka ensure experiment reproducibility?

After parsing, the framework creates an immutable `ExperimentSpec` object in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts). The spec is frozen and copied to the experiment directory alongside an append-only attempt log. Any modification requires a new spec file, guaranteeing that the frozen configuration exactly matches the executed trials.

### Can I rerun a single failed cell without re-running the entire experiment?

Yes. Use the `--cell <cell-id>` flag with the CLI command. The cell ID follows the format `task-id::repetition::subject-id`, allowing you to target specific combinations in the Cartesian product without invalidating previous results.

### How do I implement a custom executor for the Eval framework?

Implement the `ExperimentExecutor` interface defined in [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts), then pass a factory function to `runMakaEvalCli` via the `loadExecutor` property in [`packages/eval/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/cli.ts). This injects your custom logic while retaining the framework's validation, directory management, and result aggregation capabilities.