How Apache Maka Handles Evaluation and Benchmarking Semantics: A Technical Deep Dive

Apache Maka handles evaluation and benchmarking semantics through a declarative, typed experiment specification that enforces reproducible benchmark versions, orchestrates isolated Docker-based execution, and applies structured result verification across Cartesian product cells.

The Apache Maka project provides a comprehensive evaluation framework designed for rigorous, reproducible AI agent benchmarking. Understanding Maka evaluation and benchmarking semantics is essential for researchers who need to verify agent performance against version-pinned tasks while maintaining strict isolation guarantees. The system combines immutable experiment specifications with a cell-based execution model to ensure statistical validity and auditability.

Declarative Experiment Specifications

Maka’s evaluation engine begins with a typed experiment specification that declaratively defines every aspect of a benchmark run. The entry point is the parseExperimentSpec function located in [packages/eval/src/spec.ts](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts).

This parser validates top-level keys including benchmark, executor, subjects, tasks, repetitions, budget, and verifier. After validation, it freezes the resulting object to guarantee immutability throughout the experiment lifecycle. The benchmark field specifically requires three properties:

• id: The unique benchmark identifier
• version: A full Git commit hash for reproducibility
• config: Arbitrary JSON containing benchmark-specific parameters

{
  "schemaVersion": "maka.eval.v1",
  "id": "example-exp",
  "benchmark": {
    "id": "terminal-bench-2.1",
    "version": "28ebe0949f5a2c3d4e6b7c9d8f1a2b3c4d5e6f7a",
    "config": { "repository": "https://github.com/terminal-bench/terminal-bench" }
  },
  "executor": { "kind": "docker", "config": {} },
  "subjects": [{ "id": "maka", "kind": "maka", "credentials": [], "config": {} }],
  "tasks": [{ "id": "task-1", "input": "Summarize the repo", "config": {} }],
  "repetitions": 1,
  "budget": {},
  "verifier": { "reward": "pass" }
}

Reproducible Benchmark Versioning

A cornerstone of Maka’s benchmarking semantics is strict version pinning via Git commit hashes. The benchmark.version field must contain a complete SHA-1 or SHA-256 hash, validated in the decodeTask function within [packages/eval/src/harness-executor.ts](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts).

This validation ensures that every benchmark run references an exact, immutable code snapshot. The harness executor rejects abbreviated hashes or non-commit references, throwing an error if the pattern ^(?:[0-9a-f]{40}|[0-9a-f]{64})$ does not match.

// From harness-executor.ts
const revision = text(cell.benchmark.version, 'benchmark.version');
if (!/^(?:[0-9a-f]{40}|[0-9a-f]{64})$/iu.test(revision)) {
  throw new Error('Harbor benchmark.version must be a complete Git commit');
}
const task = {
  path: text(harbor.path, 'task.config.harbor.path'),
  git_url: text(benchmark.repository, 'benchmark.config.repository'),
  git_commit_id: revision,
};

Cell-Based Execution Model

Maka implements a Cartesian product execution model where an Experiment expands into discrete cells. Each cell represents a unique combination of:

• Subject: The agent or system under test
• Task: The specific benchmark task instance
• Repetition: The trial number (supporting statistical averaging)

As defined in [packages/eval/src/experiment.ts](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts), the ExperimentSpec type composes these dimensions into individual cells. Every cell receives the same frozen benchmark object, ensuring that all subject arms operate against identical benchmark metadata. This design supports fair comparison across subjects while maintaining independent failure domains for each cell.

Runtime Host and Harness Execution

The Runtime Host orchestrates the full experiment lifecycle. Upon receiving the experiment spec, it instantiates a SessionManager and spawns an AgentRun for each cell, handling timeouts, resource metering, and lifecycle management.

The HarnessExecutor materializes abstract benchmark configurations into concrete execution environments. For Harbor benchmarks, the executor extracts benchmark.config.repository and validates the commit ID. For Pier benchmarks, it enforces path containment within the declared tasksRootEnv to prevent directory traversal.

# Run the experiment from CLI

npm run cli:dev -- run "my-experiment.json"

The harness creates Docker-based isolation for each task, ensuring that benchmark environments remain consistent and that host system state does not influence results.

Result Verification and Scoring

After subject execution completes, the harness emits a RelayResultFrame containing the cell’s output. The rewardKey helper function extracts the verification criterion from cell.verifier.reward, which downstream scoring logic uses to compute pass/fail metrics and cost-per-pass calculations.

// Extract the reward key from a cell's verifier
const reward = rewardKey(cell);   // → "pass"

The evaluation layer aggregates per-cell results and applies statistical tests such as McNemar to determine significant performance differences between subjects. Final reports include reproducible metrics and are published asMarkdown documents under docs/eval/.

Isolation and Security

Benchmarks execute within an isolated egress proxy to prevent network-based contamination or data exfiltration. The system resolves network policy files from environment variables including egressProxy.composeSourceEnv and enforces these policies at runtime. This architecture ensures that benchmarks cannot access external resources beyond explicitly allowed egress rules, maintaining the integrity of isolated test environments.

Summary

• Declarative specs in [packages/eval/src/spec.ts](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) validate and freeze experiment configurations to prevent mutation during runs.
• Strict versioning requires full Git commit hashes (SHA-1/SHA-256) in benchmark.version, enforced by decodeTask in the harness executor.
• Cell model creates a Cartesian product of subjects, tasks, and repetitions, with each cell receiving identical frozen benchmark metadata.
• Docker-based isolation via the Runtime Host and HarnessExecutor ensures reproducible environments and security through egress proxies.
• Structured verification uses rewardKey extraction and RelayResultFrame aggregation to compute pass/fail rates and cost-per-pass metrics.

Frequently Asked Questions

What file format does Maka use to define benchmark experiments?

Maka uses JSON-based experiment specification files that conform to the maka.eval.v1 schema. The parseExperimentSpec function in [packages/eval/src/spec.ts](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) validates these files, requiring fields for benchmark metadata, executor configuration, subjects, tasks, and verification criteria. The parser freezes the resulting object to ensure immutability throughout the evaluation lifecycle.

Why does Maka require full Git commit hashes for benchmark versions?

Maka requires full SHA-1 or SHA-256 commit hashes to guarantee reproducibility and prevent ambiguous version references. The decodeTask function in [packages/eval/src/harness-executor.ts](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts) validates the 40-character or 64-character hex pattern, ensuring that every benchmark run references an exact, immutable code snapshot rather than a mutable branch or tag.

How does Maka ensure fair comparison between different AI subjects?

Maka ensures fair comparison through its cell-based execution model, where each subject receives the identical frozen benchmark object within its execution cell. The Runtime Host creates independent AgentRun instances for each combination of subject, task, and repetition, preventing cross-contamination while ensuring all subjects face equivalent task configurations and resource constraints.

What security measures prevent benchmarks from accessing unauthorized network resources?

Maka implements egress proxy isolation for all benchmark executions. The system loads network policies from environment variables such as egressProxy.composeSourceEnv and enforces these rules at runtime. This architecture prevents benchmark tasks from accessing external networks beyond explicitly permitted endpoints, eliminating the risk of benchmark contamination through external data sources or exfiltration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}