Apache Maka Performance Benchmarks: Terminal-Bench 2.1 Results and Evaluation Framework

Apache Maka achieves a 68.54% pass@1 rate on Terminal-Bench 2.1, outperforming the OpenCode baseline by 13.48 percentage points while maintaining equivalent cost per pass at approximately $0.0317 per successful task.

Apache Maka performance benchmarks evaluate the open-source agent harness against 89 real-world coding tasks using the Terminal-Bench 2.1 suite. The measurement framework, implemented in the packages/eval directory, provides reproducible metrics through Docker-based execution and standardized result collection.

Terminal-Bench 2.1 Benchmark Methodology

The Terminal-Bench 2.1 suite quantifies an agent’s ability to solve practical programming tasks. Apache Maka’s evaluation pipeline orchestrates these measurements through a structured workflow defined in packages/eval/src/experiment.ts.

Experiment Specification and Cell Generation

Each benchmark run initializes an ExperimentSpec defined in packages/eval/src/spec.ts. This specification binds three components: the benchmark (Terminal-Bench 2.1), the executor (Harbor or Pier), and the subjects (Maka or alternative harnesses).

The system generates experiment cells through a Cartesian product of task × repetition × subject. This creates a discrete cell for every task-subject pairing, ensuring comprehensive coverage across all 89 benchmark tasks.

Docker-based Execution Environment

The Runtime Host executes each cell inside an isolated Docker container as implemented in packages/eval/src/harness-executor.ts. Execution respects strict deadlines calculated as the task-native timeout multiplied by one, and leverages an egress-proxy that blocks benchmark-contamination URLs to prevent data leakage.

This containerized approach ensures reproducible resource constraints and network isolation across all Apache Maka performance benchmark runs.

Result Collection and Scoring Metrics

Upon completion, each cell emits a JSON result frame with the prefix MAKA-EVAL-RESULT-V1 handled by packages/eval/src/relay-result-frame.ts. The frame captures pass/fail status, token usage, cost metrics, and a SHA-256 artifact fingerprint.

The official Terminal-Bench verifier determines pass@1 scores. Apache Maka extends this with additional metrics including non-budget conditional pass rate, budget-exhaustion rate, and cost per pass calculated using DeepSeek’s frozen pricing identity ($0.145/M uncached input, $0.0029/M cached input, $0.29/M output).

Apache Maka Performance Results vs OpenCode

Benchmark results comparing Apache Maka against the OpenCode baseline using the DeepSeek V4-Flash model demonstrate superior task completion efficiency with comparable economic costs, as documented in docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md.

Pass@1 Success Rates

Apache Maka achieved 61 successful completions out of 89 tasks (68.54%), while OpenCode completed 49 tasks (55.06%). This represents a gain of 12 additional tasks or 13.48 percentage points.

The non-budget conditional pass rate—measuring success only among tasks that did not exhaust their budget—reached 85.25% for Maka versus 75.41% for OpenCode.

Budget Efficiency and Cost Analysis

Apache Maka demonstrated superior resource efficiency with a budget-exhaustion rate of 16.85% (15/89 tasks) compared to OpenCode’s 26.97% (24/89 tasks). This 10.11 percentage point improvement indicates more efficient token utilization.

The cost per pass remained effectively equivalent between implementations: $0.031718 for Maka versus $0.031712 for OpenCode, differing by only $0.000006 (0.02%). Total token consumption reached 173,281,686 for Maka compared to 115,523,680 for the baseline, with both systems achieving approximately 98% cache-hit rates on input tokens.

Statistical Significance

The performance gap is statistically significant according to an exact two-sided McNemar test yielding p = 0.0118 across 20 discordant task pairs (see docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md lines 36-38). This confirms that Apache Maka’s superior pass rate is unlikely to result from random variation.

Running the Benchmark Suite

Developers can reproduce these Apache Maka performance benchmarks using the CLI tools and configuration files provided in the repository.

CLI Execution

Install dependencies and execute the frozen Terminal-Bench 2.1 experiment:

# Install from repository root

npm ci

# Run the four-arm benchmark with Maka

maka eval run experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json \
    --out .maka-eval/run-maka-4arm

This command validates the executor configuration, launches Docker containers for each generated cell, and writes a detailed write-ahead log (WAL) under ./.maka-eval as documented in packages/eval/README.md.

Experiment Configuration

Define evaluation parameters using the ExperimentSpec schema:

{
  "schemaVersion": "maka.eval.v1",
  "id": "maka-demo",
  "benchmark": { "id": "terminal-bench-2.1", "version": "d49e28f1", "config": {} },
  "executor": { "kind": "harbor", "config": {} },
  "subjects": [{ "id": "maka", "kind": "maka", "credentials": [], "config": {} }],
  "tasks": [{ "id": "task-1", "input": "Write a Python function to reverse a string.", "config": {} }],
  "repetitions": 1,
  "budget": {},
  "verifier": {}
}

Running maka eval run experiment.json processes the single cell and outputs a result frame containing pass/fail status, token consumption, and cost data.

Parsing Result Frames

Process evaluation results programmatically using the TypeScript parser in packages/eval/src/relay-result-frame.ts:

import { parseResultFrame } from '@maka/eval/src/relay-result-frame';

const line = 'MAKA-EVAL-RESULT-V1 abcdef 1234 1a2b3c ...';
const result = parseResultFrame(line);

console.log(`Pass: ${result.payload.status === 'pass'}`);
console.log(`Cost: $${result.payload.costPerPass}`);

This utility validates result lines against the expected schema and extracts the JSON payload for further analysis.

Summary

• Apache Maka achieves 68.54% pass@1 on Terminal-Bench 2.1, significantly outperforming the OpenCode baseline’s 55.06%.
• The evaluation framework in packages/eval uses Docker-based execution with strict egress controls to ensure reproducible Apache Maka performance benchmarks.
• Cost per pass remains economically equivalent at approximately $0.0317, while budget-exhaustion rates are 10.11 percentage points lower than the baseline.
• Results are statistically significant (p = 0.0118) according to McNemar’s exact test on discordant task pairs.
• Developers can reproduce results using maka eval run with experiment specifications defined in JSON configuration files.

Frequently Asked Questions

What benchmark suite does Apache Maka use for performance testing?

Apache Maka uses Terminal-Bench 2.1, a publicly available suite of 89 real-world coding tasks. The evaluation framework is implemented in packages/eval/src/experiment.ts and supports configurable benchmarks, executors, and subjects through the ExperimentSpec interface.

How does Apache Maka's pass@1 rate compare to OpenCode?

Apache Maka achieved 68.54% pass@1 (61/89 tasks) compared to OpenCode’s 55.06% (49/89 tasks) when evaluated on the DeepSeek V4-Flash model. This 13.48 percentage point improvement is statistically significant with p = 0.0118 according to a two-sided McNemar test.

What is the cost per successful task when running Apache Maka benchmarks?

The cost per pass for Apache Maka is $0.031718, virtually identical to OpenCode’s $0.031712. This metric uses DeepSeek’s frozen pricing identity and divides total metered token usage by the number of successful task completions.

How can I reproduce the Terminal-Bench 2.1 results locally?

Execute maka eval run experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json from the repository root after running npm ci. This launches the Harbor executor inside Docker containers for each task cell and writes detailed results to ./.maka-eval as documented in packages/eval/README.md.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}