# Apache Maka Performance Benchmarks: Terminal-Bench 2.1 Results and Evaluation Framework

> Explore Apache Maka performance benchmarks with Terminal-Bench 2.1 results. Discover its impressive 68.54% pass rate and cost-effective execution.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: performance
- Published: 2026-09-11

---

**Apache Maka achieves a 68.54% pass@1 rate on Terminal-Bench 2.1, outperforming the OpenCode baseline by 13.48 percentage points while maintaining equivalent cost per pass at approximately $0.0317 per successful task.**

Apache Maka performance benchmarks evaluate the open-source agent harness against 89 real-world coding tasks using the Terminal-Bench 2.1 suite. The measurement framework, implemented in the `packages/eval` directory, provides reproducible metrics through Docker-based execution and standardized result collection.

## Terminal-Bench 2.1 Benchmark Methodology

The Terminal-Bench 2.1 suite quantifies an agent’s ability to solve practical programming tasks. Apache Maka’s evaluation pipeline orchestrates these measurements through a structured workflow defined in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts).

### Experiment Specification and Cell Generation

Each benchmark run initializes an `ExperimentSpec` defined in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts). This specification binds three components: the benchmark (Terminal-Bench 2.1), the executor (Harbor or Pier), and the subjects (Maka or alternative harnesses).

The system generates experiment cells through a Cartesian product of `task × repetition × subject`. This creates a discrete cell for every task-subject pairing, ensuring comprehensive coverage across all 89 benchmark tasks.

### Docker-based Execution Environment

The **Runtime Host** executes each cell inside an isolated Docker container as implemented in [`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts). Execution respects strict deadlines calculated as the task-native timeout multiplied by one, and leverages an egress-proxy that blocks benchmark-contamination URLs to prevent data leakage.

This containerized approach ensures reproducible resource constraints and network isolation across all Apache Maka performance benchmark runs.

### Result Collection and Scoring Metrics

Upon completion, each cell emits a JSON result frame with the prefix `MAKA-EVAL-RESULT-V1` handled by [`packages/eval/src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/relay-result-frame.ts). The frame captures pass/fail status, token usage, cost metrics, and a SHA-256 artifact fingerprint.

The official Terminal-Bench verifier determines **pass@1** scores. Apache Maka extends this with additional metrics including **non-budget conditional pass rate**, **budget-exhaustion rate**, and **cost per pass** calculated using DeepSeek’s frozen pricing identity ($0.145/M uncached input, $0.0029/M cached input, $0.29/M output).

## Apache Maka Performance Results vs OpenCode

Benchmark results comparing Apache Maka against the OpenCode baseline using the DeepSeek V4-Flash model demonstrate superior task completion efficiency with comparable economic costs, as documented in [`docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md`](https://github.com/apache/maka/blob/main/docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md).

### Pass@1 Success Rates

Apache Maka achieved **61 successful completions out of 89 tasks (68.54%)**, while OpenCode completed 49 tasks (55.06%). This represents a gain of 12 additional tasks or 13.48 percentage points.

The **non-budget conditional pass rate**—measuring success only among tasks that did not exhaust their budget—reached 85.25% for Maka versus 75.41% for OpenCode.

### Budget Efficiency and Cost Analysis

Apache Maka demonstrated superior resource efficiency with a **budget-exhaustion rate of 16.85%** (15/89 tasks) compared to OpenCode’s 26.97% (24/89 tasks). This 10.11 percentage point improvement indicates more efficient token utilization.

The **cost per pass** remained effectively equivalent between implementations: **$0.031718** for Maka versus $0.031712 for OpenCode, differing by only $0.000006 (0.02%). Total token consumption reached 173,281,686 for Maka compared to 115,523,680 for the baseline, with both systems achieving approximately 98% cache-hit rates on input tokens.

### Statistical Significance

The performance gap is statistically significant according to an exact two-sided **McNemar test** yielding **p = 0.0118** across 20 discordant task pairs (see [`docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md`](https://github.com/apache/maka/blob/main/docs/eval/terminal-bench-2.1-deepseek-v4-flash-maka-vs-opencode.md) lines 36-38). This confirms that Apache Maka’s superior pass rate is unlikely to result from random variation.

## Running the Benchmark Suite

Developers can reproduce these Apache Maka performance benchmarks using the CLI tools and configuration files provided in the repository.

### CLI Execution

Install dependencies and execute the frozen Terminal-Bench 2.1 experiment:

```bash

# Install from repository root

npm ci

# Run the four-arm benchmark with Maka

maka eval run experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json \
    --out .maka-eval/run-maka-4arm

```

This command validates the executor configuration, launches Docker containers for each generated cell, and writes a detailed write-ahead log (WAL) under `./.maka-eval` as documented in [`packages/eval/README.md`](https://github.com/apache/maka/blob/main/packages/eval/README.md).

### Experiment Configuration

Define evaluation parameters using the `ExperimentSpec` schema:

```json
{
  "schemaVersion": "maka.eval.v1",
  "id": "maka-demo",
  "benchmark": { "id": "terminal-bench-2.1", "version": "d49e28f1", "config": {} },
  "executor": { "kind": "harbor", "config": {} },
  "subjects": [{ "id": "maka", "kind": "maka", "credentials": [], "config": {} }],
  "tasks": [{ "id": "task-1", "input": "Write a Python function to reverse a string.", "config": {} }],
  "repetitions": 1,
  "budget": {},
  "verifier": {}
}

```

Running `maka eval run experiment.json` processes the single cell and outputs a result frame containing pass/fail status, token consumption, and cost data.

### Parsing Result Frames

Process evaluation results programmatically using the TypeScript parser in [`packages/eval/src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/relay-result-frame.ts):

```typescript
import { parseResultFrame } from '@maka/eval/src/relay-result-frame';

const line = 'MAKA-EVAL-RESULT-V1 abcdef 1234 1a2b3c ...';
const result = parseResultFrame(line);

console.log(`Pass: ${result.payload.status === 'pass'}`);
console.log(`Cost: $${result.payload.costPerPass}`);

```

This utility validates result lines against the expected schema and extracts the JSON payload for further analysis.

## Summary

- Apache Maka achieves **68.54% pass@1** on Terminal-Bench 2.1, significantly outperforming the OpenCode baseline’s 55.06%.
- The evaluation framework in `packages/eval` uses Docker-based execution with strict egress controls to ensure reproducible Apache Maka performance benchmarks.
- **Cost per pass** remains economically equivalent at approximately $0.0317, while **budget-exhaustion rates** are 10.11 percentage points lower than the baseline.
- Results are statistically significant (p = 0.0118) according to McNemar’s exact test on discordant task pairs.
- Developers can reproduce results using `maka eval run` with experiment specifications defined in JSON configuration files.

## Frequently Asked Questions

### What benchmark suite does Apache Maka use for performance testing?

Apache Maka uses **Terminal-Bench 2.1**, a publicly available suite of 89 real-world coding tasks. The evaluation framework is implemented in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts) and supports configurable benchmarks, executors, and subjects through the `ExperimentSpec` interface.

### How does Apache Maka's pass@1 rate compare to OpenCode?

Apache Maka achieved **68.54% pass@1** (61/89 tasks) compared to OpenCode’s **55.06%** (49/89 tasks) when evaluated on the DeepSeek V4-Flash model. This 13.48 percentage point improvement is statistically significant with p = 0.0118 according to a two-sided McNemar test.

### What is the cost per successful task when running Apache Maka benchmarks?

The **cost per pass** for Apache Maka is **$0.031718**, virtually identical to OpenCode’s $0.031712. This metric uses DeepSeek’s frozen pricing identity and divides total metered token usage by the number of successful task completions.

### How can I reproduce the Terminal-Bench 2.1 results locally?

Execute `maka eval run experiments/terminal-bench-2.1-deepseek-v4-flash-four-arm.json` from the repository root after running `npm ci`. This launches the Harbor executor inside Docker containers for each task cell and writes detailed results to `./.maka-eval` as documented in [`packages/eval/README.md`](https://github.com/apache/maka/blob/main/packages/eval/README.md).