# Mako Evaluation Framework: Understanding Experiments, Cells, and Attempts

> Master Mako's evaluation framework. Understand Experiments, Cells, and Attempts for reproducible benchmark runs. Learn the core abstractions of apache/maka.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-06

---

**Mako's evaluation framework (`@maka/eval`) uses four core abstractions—Experiment, Cell, Attempt, and Result—to model reproducible, append-only benchmark runs.**

The Apache Maka project provides a structured evaluation system for benchmarking AI agents and subjects. At its heart are three foundational concepts—**Experiment**, **Cell**, and **Attempt**—that transform declarative specifications into deterministic execution units with immutable records. This article explains how these concepts work together based on the official source code in `apache/maka`.

## Experiment: The Declarative Specification

An **Experiment** is a frozen, declarative spec that binds together every component needed for evaluation.

In [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts) (lines 46–70), the `ExperimentSpec` interface defines the complete structure:

- A **benchmark** (what is being evaluated)
- An **executor** (how subjects run)
- **Subjects** (the agents or systems under test)
- **Tasks** (specific inputs or prompts)
- **Repetitions** (how many times to run each task)
- **Budget**, **verifier**, and concurrency limits

The spec is parsed and **deep-frozen** by `parseExperimentSpec` in `spec.ts:22`, ensuring immutability throughout execution. Validation catches misconfigurations early, before any resources are allocated.

```json
{
  "schemaVersion": "maka.eval.v1",
  "id": "demo-experiment",
  "benchmark": { "id": "term-bench", "version": "2.0", "config": {} },
  "executor": { "kind": "harbor", "config": {} },
  "execution": { "maxConcurrentTaskGroups": 2 },
  "subjects": [
    { "id": "maka-subject", "kind": "maka", "credentials": [], "config": {} }
  ],
  "tasks": [
    { "id": "task-01", "input": "Explain recursion.", "config": {} }
  ],
  "repetitions": 3,
  "budget": {},
  "verifier": {}
}

```

## Cell: The Cartesian Product of Execution

A **Cell** represents a single logical execution unit—the Cartesian product of **task × repetition × subject**.

The `expandExperiment` function in `experiment.ts:84–99` walks the spec and generates all `ExperimentCell` objects. Each cell receives a deterministic identifier:

```

<taskId>::<repetition>::<subjectId>

```

For the example above with 1 task, 3 repetitions, and 1 subject, `expandExperiment` produces three cells:

- `task-01::1::maka-subject`
- `task-01::2::maka-subject`
- `task-01::3::maka-subject`

This expansion guarantees that every possible combination is explicitly enumerated before execution begins. No dynamic discovery occurs during the run, making the evaluation fully reproducible.

## Attempt: Immutable Execution Records

An **Attempt** is an append-only record of one run of a cell. Attempts are written by the executor and **never overwritten**.

According to [`packages/eval/README.md`](https://github.com/apache/maka/blob/main/packages/eval/README.md) (line 43), the `FileAttemptStore` creates a new attempt file per cell. A leftover `.writer.lock` file indicates an unfinished write, enabling crash recovery without data corruption.

The attempt lifecycle follows this pattern:

1. **Executor prepares** the cell environment (e.g., Docker container via Harbor/Pier)
2. **`runAttempt` is called** with the cell and a callback that performs the actual work
3. **Result is wrapped** in a relay result frame and appended to storage
4. **Earliest successful attempt** is used for result selection

This append-only design means failed or partial attempts remain visible for debugging. Re-running a specific cell creates a new attempt file rather than mutating history.

## Result: The Lightweight Outcome Kernel

The **Result** abstraction strips away execution noise, preserving only comparable metrics.

As defined in `relay-result-frame.ts:24`, the result kernel contains:

| Field | Purpose |
|-------|---------|
| `score` | Evaluation score (primary outcome) |
| `normalizedUsage` | Resource consumption metrics |
| `cost` | Financial or compute cost |
| `duration` | Wall-clock execution time |
| `status` | Success/failure state |
| `artifacts` | References to external outputs |

Raw stdout, logs, and other metadata are stored as separate artifacts—not embedded in the result. This keeps the result kernel lightweight and enables fair comparison across different executors or hardware configurations.

## Executor and Subject Adapter Architecture

Two interfaces—`ExperimentExecutor` and `SubjectAdapter`—determine how cells become attempts.

**ExperimentExecutor** (`runner.ts:74–130`) controls *how* attempts run:
- Harbor or Pier for Docker-based isolation
- Custom executors for bare-metal or cloud environments

**SubjectAdapter** controls *what* the subject does:
- Maka-native subjects with full SDK integration
- External commands via shell invocation

Both interfaces expose optional hooks: `validate`, `prepare`, `canReuse`, and `execute`. A custom executor implementing `runAttempt` can be injected via `EvalCliDependencies` (`cli.ts:11–18`).

```typescript
import { ExperimentExecutor, ExperimentCell } from '@maka/eval';

export const myExecutor: ExperimentExecutor = {
  kind: 'my-exec',
  async runAttempt({ cell }, operation) {
    const result = await operation({
      context: {/* subject context */},
      verify: async () => ({ /* verification data */ })
    });
    return { status: 'ok', result };
  },
};

```

## Running Experiments via CLI

The `maka eval` command orchestrates the full flow:

```bash
maka eval run experiment.json --out ./run-001

```

The CLI driver (`cli.ts:34–42`) performs these steps:

1. Parses and validates the JSON spec
2. Checks executor prerequisites (e.g., Docker availability)
3. Expands the spec into cells
4. Launches attempts respecting `maxConcurrentTaskGroups`
5. Collects results into the output directory

For targeted re-execution, use `--cell <cell-id>` to replace a failed cell without rerunning the entire experiment.

## Summary

- **Experiment**: A frozen, validated specification binding benchmark, executor, subjects, tasks, and repetitions together (`experiment.ts:46–70`)
- **Cell**: A deterministic execution unit identified as `task::repetition::subject`, generated by `expandExperiment` (`experiment.ts:84–99`)
- **Attempt**: An immutable, append-only record of one cell execution, stored via `FileAttemptStore` with crash-safe `.writer.lock` files
- **Result**: A lightweight kernel of essential metrics (score, cost, duration, status) with external artifact references (`relay-result-frame.ts:24`)
- **CLI**: `maka eval run` validates, expands, and executes experiments with per-cell retry support ([`cli.ts`](https://github.com/apache/maka/blob/main/cli.ts))

## Frequently Asked Questions

### How does Mako ensure reproducibility across evaluation runs?

Mako guarantees reproducibility through deep-frozen specs, deterministic cell identifiers, and append-only attempt records. The `parseExperimentSpec` function in `spec.ts:71,146–152` freezes the configuration after validation, preventing mutation during execution. Cell IDs are generated purely from spec content, not runtime state.

### What happens if an attempt crashes mid-execution?

The `FileAttemptStore` leaves a `.writer.lock` file when a write begins but doesn't complete. On restart, the framework detects this orphan lock and treats the attempt as unfinished, allowing clean retry without data corruption. The original partial data remains for forensic analysis.

### Can I use a custom container runtime instead of Docker?

Yes. Implement the `ExperimentExecutor` interface from `runner.ts:74–130` and inject your implementation via `EvalCliDependencies` (`cli.ts:11–18`). The `runAttempt` method receives the cell specification and a callback—you control the environment preparation, execution, and result capture.

### How do I re-run only failed cells without starting over?

Pass `--cell <cell-id>` to the CLI, where `<cell-id>` uses the format `taskId::repetition::subjectId`. This creates a new attempt file for that specific cell while preserving all other attempts. The evaluation result automatically selects the earliest successful attempt across all retries.