How to Configure an Experiment in Maka’s Eval Framework: Complete Guide with Examples

Configure a Maka experiment by writing an experiment.json spec that defines your benchmark, executor, subjects, and tasks, then execute it via maka eval run experiment.json --out <dir>.

The Apache Maka repository provides a rigorous Eval framework for reproducible AI benchmarking. To configure an experiment in Maka's Eval Framework, you create a declarative JSON specification that the system validates against a hand-written decoder, freezes for immutability, and expands into a Cartesian product of execution cells. This guide walks through the spec structure, validation logic, and execution flow using actual source paths from the codebase.

Understanding the Experiment Specification Format

The experiment specification is a JSON file that completely describes how to run a benchmark. According to the source code in packages/eval/src/spec.ts, the framework uses a hand-written decoder—not an external JSON Schema library—to parse and validate the spec, ensuring strict type safety and explicit versioning.

Required Schema Fields

Every experiment.json must include these top-level keys:

• schemaVersion: Must be the exact string "maka.eval.v1". Unknown top-level keys cause immediate validation errors.
• id: A human-readable string identifier for the experiment.
• benchmark: An object containing id, version, and an opaque config object for benchmark-specific settings.
• executor: Specifies the execution backend with kind (e.g., "harbor" or "pier") and config.
• subjects: An array of subject definitions, each with id, kind ("maka" or "external"), optional credentials, and config.
• tasks: An array of task objects, each requiring id, input (the prompt or payload), and config.
• repetitions: A positive integer specifying how many times to sample each cell.
• budget: Resource limits passed as a JsonObject to the executor.
• verifier: Benchmark-specific verification logic as a JsonObject.

Optional Execution Parameters

The execution field accepts properties like maxConcurrentTaskGroups to limit parallelism. If omitted, the framework defaults to single-threaded execution.

Step-by-Step Configuration Guide

Step 1: Define the Benchmark and Executor

Start your experiment.json by specifying which benchmark to run and which executor will run it. In packages/eval/src/spec.ts, the decoder validates that the benchmark.id and executor.kind are present and correctly typed.

{
  "schemaVersion": "maka.eval.v1",
  "id": "terminal-benchmark-v2",
  "benchmark": {
    "id": "terminal-bench",
    "version": "2.1",
    "config": {}
  },
  "executor": {
    "kind": "harbor",
    "config": {}
  }
}

Step 2: Configure Subjects and Tasks

Subjects are the models or systems under test. Tasks define the inputs. The framework expands these into a Cartesian product in packages/eval/src/experiment.ts, creating cells for every combination of task × subject × repetition.

{
  "subjects": [
    {
      "id": "maka-subject",
      "kind": "maka",
      "credentials": [],
      "config": {}
    }
  ],
  "tasks": [
    {
      "id": "task-1",
      "input": "Write a short poem about clouds.",
      "config": {}
    }
  ]
}

Step 3: Set Repetitions and Budget

Control statistical robustness and resource consumption:

{
  "repetitions": 3,
  "budget": {
    "maxSteps": 5000
  },
  "verifier": {
    "type": "exact-match",
    "expected": ".*"
  }
}

The repetitions value directly determines how many samples the runner collects per cell before aggregating results.

Running the Experiment

Once configured, execute via the CLI entry point defined in packages/cli/src/cli-core.ts:

# Create output directory

mkdir -p .maka-eval/run-001

# Run the experiment

maka eval run experiment.json --out .maka-eval/run-001

The CLI performs prerequisite checks—Docker images, relay files, environment variables—before starting any trials. If a prerequisite is missing, it aborts immediately.

Rerunning Individual Cells

Use the --cell <cell-id> flag to replace a single failed cell without re-running the entire experiment:

maka eval run experiment.json --out .maka-eval/run-001 --cell "task-1::1::maka-subject"

Extending the Framework with Custom Executors

For advanced use cases, implement the ExperimentExecutor or SubjectAdapter interfaces from packages/eval/src/runner.ts and inject them via EvalCliDependencies in packages/eval/src/cli.ts.

import { runMakaEvalCli } from '@maka/eval';
import { createMyExecutor } from './my-executor';

runMakaEvalCli({
  specPath: 'experiment.json',
  outPath: '.maka-eval/run-custom',
  loadExecutor: (spec, specPath) => createMyExecutor(spec, specPath),
  subjects: [],
  stdout: console.log,
  stderr: console.error,
});

This allows integration with proprietary schedulers or custom model endpoints while retaining the framework's validation, directory management in packages/eval/src/experiment-directory.ts, and result aggregation.

Summary

• Configure an experiment in Maka's Eval Framework by creating an experiment.json with schemaVersion: "maka.eval.v1" and defining benchmark, executor, subjects, tasks, repetitions, budget, and verifier.
• Validation occurs through a strict hand-written decoder in packages/eval/src/spec.ts that rejects unknown keys.
• Immutability is enforced when the ExperimentSpec is frozen in packages/eval/src/experiment.ts, ensuring reproducibility.
• Execution expands the spec into cells (task × repetition × subject) and persists results to an append-only log managed by packages/eval/src/experiment-directory.ts.
• Customization is supported via the EvalCliDependencies interface for injecting custom executors or subjects.

Frequently Asked Questions

What is the required schema version for Maka experiment specs?

The schemaVersion field must be exactly "maka.eval.v1". The hand-written decoder in packages/eval/src/spec.ts enforces this string literal and rejects unknown top-level keys to prevent configuration drift.

How does Maka ensure experiment reproducibility?

After parsing, the framework creates an immutable ExperimentSpec object in packages/eval/src/experiment.ts. The spec is frozen and copied to the experiment directory alongside an append-only attempt log. Any modification requires a new spec file, guaranteeing that the frozen configuration exactly matches the executed trials.

Can I rerun a single failed cell without re-running the entire experiment?

Yes. Use the --cell <cell-id> flag with the CLI command. The cell ID follows the format task-id::repetition::subject-id, allowing you to target specific combinations in the Cartesian product without invalidating previous results.

How do I implement a custom executor for the Eval framework?

Implement the ExperimentExecutor interface defined in packages/eval/src/runner.ts, then pass a factory function to runMakaEvalCli via the loadExecutor property in packages/eval/src/cli.ts. This injects your custom logic while retaining the framework's validation, directory management, and result aggregation capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}