Mako Evaluation Framework: Understanding Experiments, Cells, and Attempts

Mako's evaluation framework (@maka/eval) uses four core abstractions—Experiment, Cell, Attempt, and Result—to model reproducible, append-only benchmark runs.

The Apache Maka project provides a structured evaluation system for benchmarking AI agents and subjects. At its heart are three foundational concepts—Experiment, Cell, and Attempt—that transform declarative specifications into deterministic execution units with immutable records. This article explains how these concepts work together based on the official source code in apache/maka.

Experiment: The Declarative Specification

An Experiment is a frozen, declarative spec that binds together every component needed for evaluation.

In packages/eval/src/experiment.ts (lines 46–70), the ExperimentSpec interface defines the complete structure:

• A benchmark (what is being evaluated)
• An executor (how subjects run)
• Subjects (the agents or systems under test)
• Tasks (specific inputs or prompts)
• Repetitions (how many times to run each task)
• Budget, verifier, and concurrency limits

The spec is parsed and deep-frozen by parseExperimentSpec in spec.ts:22, ensuring immutability throughout execution. Validation catches misconfigurations early, before any resources are allocated.

{
  "schemaVersion": "maka.eval.v1",
  "id": "demo-experiment",
  "benchmark": { "id": "term-bench", "version": "2.0", "config": {} },
  "executor": { "kind": "harbor", "config": {} },
  "execution": { "maxConcurrentTaskGroups": 2 },
  "subjects": [
    { "id": "maka-subject", "kind": "maka", "credentials": [], "config": {} }
  ],
  "tasks": [
    { "id": "task-01", "input": "Explain recursion.", "config": {} }
  ],
  "repetitions": 3,
  "budget": {},
  "verifier": {}
}

Cell: The Cartesian Product of Execution

A Cell represents a single logical execution unit—the Cartesian product of task × repetition × subject.

The expandExperiment function in experiment.ts:84–99 walks the spec and generates all ExperimentCell objects. Each cell receives a deterministic identifier:

<taskId>::<repetition>::<subjectId>

For the example above with 1 task, 3 repetitions, and 1 subject, expandExperiment produces three cells:

• task-01::1::maka-subject
• task-01::2::maka-subject
• task-01::3::maka-subject

This expansion guarantees that every possible combination is explicitly enumerated before execution begins. No dynamic discovery occurs during the run, making the evaluation fully reproducible.

Attempt: Immutable Execution Records

An Attempt is an append-only record of one run of a cell. Attempts are written by the executor and never overwritten.

According to packages/eval/README.md (line 43), the FileAttemptStore creates a new attempt file per cell. A leftover .writer.lock file indicates an unfinished write, enabling crash recovery without data corruption.

The attempt lifecycle follows this pattern:

1. Executor prepares the cell environment (e.g., Docker container via Harbor/Pier)
2. runAttempt is called with the cell and a callback that performs the actual work
3. Result is wrapped in a relay result frame and appended to storage
4. Earliest successful attempt is used for result selection

This append-only design means failed or partial attempts remain visible for debugging. Re-running a specific cell creates a new attempt file rather than mutating history.

Result: The Lightweight Outcome Kernel

The Result abstraction strips away execution noise, preserving only comparable metrics.

As defined in relay-result-frame.ts:24, the result kernel contains:

Field	Purpose
score	Evaluation score (primary outcome)
normalizedUsage	Resource consumption metrics
cost	Financial or compute cost
duration	Wall-clock execution time
status	Success/failure state
artifacts	References to external outputs

Raw stdout, logs, and other metadata are stored as separate artifacts—not embedded in the result. This keeps the result kernel lightweight and enables fair comparison across different executors or hardware configurations.

Executor and Subject Adapter Architecture

Two interfaces—ExperimentExecutor and SubjectAdapter—determine how cells become attempts.

ExperimentExecutor (runner.ts:74–130) controls how attempts run:

• Harbor or Pier for Docker-based isolation
• Custom executors for bare-metal or cloud environments

SubjectAdapter controls what the subject does:

• Maka-native subjects with full SDK integration
• External commands via shell invocation

Both interfaces expose optional hooks: validate, prepare, canReuse, and execute. A custom executor implementing runAttempt can be injected via EvalCliDependencies (cli.ts:11–18).

import { ExperimentExecutor, ExperimentCell } from '@maka/eval';

export const myExecutor: ExperimentExecutor = {
  kind: 'my-exec',
  async runAttempt({ cell }, operation) {
    const result = await operation({
      context: {/* subject context */},
      verify: async () => ({ /* verification data */ })
    });
    return { status: 'ok', result };
  },
};

Running Experiments via CLI

The maka eval command orchestrates the full flow:

maka eval run experiment.json --out ./run-001

The CLI driver (cli.ts:34–42) performs these steps:

1. Parses and validates the JSON spec
2. Checks executor prerequisites (e.g., Docker availability)
3. Expands the spec into cells
4. Launches attempts respecting maxConcurrentTaskGroups
5. Collects results into the output directory

For targeted re-execution, use --cell <cell-id> to replace a failed cell without rerunning the entire experiment.

Summary

• Experiment: A frozen, validated specification binding benchmark, executor, subjects, tasks, and repetitions together (experiment.ts:46–70)
• Cell: A deterministic execution unit identified as task::repetition::subject, generated by expandExperiment (experiment.ts:84–99)
• Attempt: An immutable, append-only record of one cell execution, stored via FileAttemptStore with crash-safe .writer.lock files
• Result: A lightweight kernel of essential metrics (score, cost, duration, status) with external artifact references (relay-result-frame.ts:24)
• CLI: maka eval run validates, expands, and executes experiments with per-cell retry support (cli.ts)

Frequently Asked Questions

How does Mako ensure reproducibility across evaluation runs?

Mako guarantees reproducibility through deep-frozen specs, deterministic cell identifiers, and append-only attempt records. The parseExperimentSpec function in spec.ts:71,146–152 freezes the configuration after validation, preventing mutation during execution. Cell IDs are generated purely from spec content, not runtime state.

What happens if an attempt crashes mid-execution?

The FileAttemptStore leaves a .writer.lock file when a write begins but doesn't complete. On restart, the framework detects this orphan lock and treats the attempt as unfinished, allowing clean retry without data corruption. The original partial data remains for forensic analysis.

Can I use a custom container runtime instead of Docker?

Yes. Implement the ExperimentExecutor interface from runner.ts:74–130 and inject your implementation via EvalCliDependencies (cli.ts:11–18). The runAttempt method receives the cell specification and a callback—you control the environment preparation, execution, and result capture.

How do I re-run only failed cells without starting over?

Pass --cell <cell-id> to the CLI, where <cell-id> uses the format taskId::repetition::subjectId. This creates a new attempt file for that specific cell while preserving all other attempts. The evaluation result automatically selects the earliest successful attempt across all retries.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}