# How Apache Maka Handles Evaluation and Benchmarking Semantics: A Technical Deep Dive

> Explore how Apache Maka handles evaluation and benchmarking semantics. Discover reproducible versions, isolated Docker execution, and structured result verification. Learn more!

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-12

---

**Apache Maka handles evaluation and benchmarking semantics through a declarative, typed experiment specification that enforces reproducible benchmark versions, orchestrates isolated Docker-based execution, and applies structured result verification across Cartesian product cells.**

The Apache Maka project provides a comprehensive evaluation framework designed for rigorous, reproducible AI agent benchmarking. Understanding Maka evaluation and benchmarking semantics is essential for researchers who need to verify agent performance against version-pinned tasks while maintaining strict isolation guarantees. The system combines immutable experiment specifications with a cell-based execution model to ensure statistical validity and auditability.

## Declarative Experiment Specifications

Maka’s evaluation engine begins with a **typed experiment specification** that declaratively defines every aspect of a benchmark run. The entry point is the `parseExperimentSpec` function located in [[`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts).

This parser validates top-level keys including `benchmark`, `executor`, `subjects`, `tasks`, `repetitions`, `budget`, and `verifier`. After validation, it **freezes the resulting object** to guarantee immutability throughout the experiment lifecycle. The `benchmark` field specifically requires three properties:

- `id`: The unique benchmark identifier
- `version`: A full Git commit hash for reproducibility
- `config`: Arbitrary JSON containing benchmark-specific parameters

```ts
{
  "schemaVersion": "maka.eval.v1",
  "id": "example-exp",
  "benchmark": {
    "id": "terminal-bench-2.1",
    "version": "28ebe0949f5a2c3d4e6b7c9d8f1a2b3c4d5e6f7a",
    "config": { "repository": "https://github.com/terminal-bench/terminal-bench" }
  },
  "executor": { "kind": "docker", "config": {} },
  "subjects": [{ "id": "maka", "kind": "maka", "credentials": [], "config": {} }],
  "tasks": [{ "id": "task-1", "input": "Summarize the repo", "config": {} }],
  "repetitions": 1,
  "budget": {},
  "verifier": { "reward": "pass" }
}

```

## Reproducible Benchmark Versioning

A cornerstone of Maka’s benchmarking semantics is **strict version pinning** via Git commit hashes. The `benchmark.version` field must contain a complete SHA-1 or SHA-256 hash, validated in the `decodeTask` function within [[`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts).

This validation ensures that every benchmark run references an exact, immutable code snapshot. The harness executor rejects abbreviated hashes or non-commit references, throwing an error if the pattern `^(?:[0-9a-f]{40}|[0-9a-f]{64})$` does not match.

```ts
// From harness-executor.ts
const revision = text(cell.benchmark.version, 'benchmark.version');
if (!/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/iu.test(revision)) {
  throw new Error('Harbor benchmark.version must be a complete Git commit');
}
const task = {
  path: text(harbor.path, 'task.config.harbor.path'),
  git_url: text(benchmark.repository, 'benchmark.config.repository'),
  git_commit_id: revision,
};

```

## Cell-Based Execution Model

Maka implements a **Cartesian product execution model** where an *Experiment* expands into discrete *cells*. Each cell represents a unique combination of:

- **Subject**: The agent or system under test
- **Task**: The specific benchmark task instance
- **Repetition**: The trial number (supporting statistical averaging)

As defined in [[`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), the `ExperimentSpec` type composes these dimensions into individual cells. Every cell receives the same **frozen benchmark object**, ensuring that all subject arms operate against identical benchmark metadata. This design supports fair comparison across subjects while maintaining independent failure domains for each cell.

## Runtime Host and Harness Execution

The **Runtime Host** orchestrates the full experiment lifecycle. Upon receiving the experiment spec, it instantiates a `SessionManager` and spawns an `AgentRun` for each cell, handling timeouts, resource metering, and lifecycle management.

The **HarnessExecutor** materializes abstract benchmark configurations into concrete execution environments. For *Harbor* benchmarks, the executor extracts `benchmark.config.repository` and validates the commit ID. For *Pier* benchmarks, it enforces path containment within the declared `tasksRootEnv` to prevent directory traversal.

```bash

# Run the experiment from CLI

npm run cli:dev -- run "my-experiment.json"

```

The harness creates Docker-based isolation for each task, ensuring that benchmark environments remain consistent and that host system state does not influence results.

## Result Verification and Scoring

After subject execution completes, the harness emits a **RelayResultFrame** containing the cell’s output. The `rewardKey` helper function extracts the verification criterion from `cell.verifier.reward`, which downstream scoring logic uses to compute **pass/fail metrics** and **cost-per-pass calculations**.

```ts
// Extract the reward key from a cell's verifier
const reward = rewardKey(cell);   // → "pass"

```

The evaluation layer aggregates per-cell results and applies statistical tests such as McNemar to determine significant performance differences between subjects. Final reports include reproducible metrics and are published asMarkdown documents under `docs/eval/`.

## Isolation and Security

Benchmarks execute within an **isolated egress proxy** to prevent network-based contamination or data exfiltration. The system resolves network policy files from environment variables including `egressProxy.composeSourceEnv` and enforces these policies at runtime. This architecture ensures that benchmarks cannot access external resources beyond explicitly allowed egress rules, maintaining the integrity of isolated test environments.

## Summary

- **Declarative specs** in [[`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) validate and freeze experiment configurations to prevent mutation during runs.
- **Strict versioning** requires full Git commit hashes (SHA-1/SHA-256) in `benchmark.version`, enforced by `decodeTask` in the harness executor.
- **Cell model** creates a Cartesian product of subjects, tasks, and repetitions, with each cell receiving identical frozen benchmark metadata.
- **Docker-based isolation** via the Runtime Host and HarnessExecutor ensures reproducible environments and security through egress proxies.
- **Structured verification** uses `rewardKey` extraction and RelayResultFrame aggregation to compute pass/fail rates and cost-per-pass metrics.

## Frequently Asked Questions

### What file format does Maka use to define benchmark experiments?

Maka uses JSON-based experiment specification files that conform to the `maka.eval.v1` schema. The `parseExperimentSpec` function in [[`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) validates these files, requiring fields for benchmark metadata, executor configuration, subjects, tasks, and verification criteria. The parser freezes the resulting object to ensure immutability throughout the evaluation lifecycle.

### Why does Maka require full Git commit hashes for benchmark versions?

Maka requires full SHA-1 or SHA-256 commit hashes to guarantee **reproducibility** and prevent ambiguous version references. The `decodeTask` function in [[`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts)](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts) validates the 40-character or 64-character hex pattern, ensuring that every benchmark run references an exact, immutable code snapshot rather than a mutable branch or tag.

### How does Maka ensure fair comparison between different AI subjects?

Maka ensures fair comparison through its **cell-based execution model**, where each subject receives the identical frozen `benchmark` object within its execution cell. The Runtime Host creates independent `AgentRun` instances for each combination of subject, task, and repetition, preventing cross-contamination while ensuring all subjects face equivalent task configurations and resource constraints.

### What security measures prevent benchmarks from accessing unauthorized network resources?

Maka implements **egress proxy isolation** for all benchmark executions. The system loads network policies from environment variables such as `egressProxy.composeSourceEnv` and enforces these rules at runtime. This architecture prevents benchmark tasks from accessing external networks beyond explicitly permitted endpoints, eliminating the risk of benchmark contamination through external data sources or exfiltration.