How Apache Maka Manages Experiments and Evaluations: A Technical Deep Dive

Apache Maka treats experiments and evaluations as first-class assets described by JSON specifications and executed through a dedicated evaluation framework in packages/eval that validates schemas, manages concurrency, and aggregates results across distributed runtime hosts.

Apache Maka provides a robust framework for managing experiments and evaluations through a declarative, version-controlled approach. The system treats each experiment as a structured asset defined by JSON specifications and orchestrated by a dedicated evaluation engine located in the packages/eval directory. This architecture ensures reproducible, isolated test execution across diverse runtime environments while maintaining strict consistency between on-disk configurations and in-memory representations.

Experiment Definition and Schema Validation

Every experiment in Maka lives under the top-level experiments/ folder and is defined by a specification file named experiment.json. This JSON file declares the experiment ID, a list of subjects (the services or modules under test), and the tasks (actions each subject should perform).

The schema is enforced by packages/eval/src/experiment.ts, which validates the schemaVersion field (expected value: maka.eval.v1) and guarantees that every experiment contains at least one subject and one task. This validation ensures structural integrity before any execution begins.

// Example experiment.json structure validated by the framework
{
  "schemaVersion": "maka.eval.v1",
  "id": "terminal-bench-2.1-deepseek-v4-flash-eight-arm",
  "subjects": ["service-a", "service-b"],
  "tasks": ["benchmark", "validate"]
}

Directory Handling and Spec Consistency

Before execution, the framework ensures the on-disk specification matches the expected in-memory representation. The packages/eval/src/experiment-directory.ts module opens the experiment directory, reads the experiment.json file, and performs strict consistency checks.

If the specification differs from the expected structure or version, the framework throws an error immediately. This prevents silent configuration drift and ensures that experiments remain reproducible across different execution environments.

The Evaluation Execution Pipeline

Running an experiment involves two core components that transform specifications into executable units. First, packages/eval/src/runner.ts parses the specification and builds a cell for each (subject, task) pair, creating a matrix of individual test units.

These cells are then handed to the harness executor (packages/eval/src/harness-executor.ts), which delegates execution to the appropriate runtime environment. Supported runtimes include Docker containers, Windows sandboxes, and remote hosts, allowing flexible deployment across heterogeneous infrastructure.

// Loading and running an experiment using the Maka evaluation API
import { openExperimentDirectory, runExperiment } from '@maka/eval';

// 1. Load the experiment directory
const expDir = await openExperimentDirectory(
  new URL('../experiments/terminal-bench-2.1-deepseek-v4-flash-eight-arm.json', import.meta.url)
);

// 2. Execute the experiment
const result = await runExperiment(expDir);

// 3. Access results
console.log(`Experiment ${result.experimentId} completed ${result.cells} cells`);

Result Collection and CLI Reporting

Once execution completes, packages/eval/src/result.ts aggregates outcomes into a structured result object. This object contains the experimentId, the total count of completed cells, and a count of any incomplete cells that failed or timed out.

The command-line interface (packages/eval/src/cli.ts) consumes these results and prints a concise JSON line summarizing the run, following this format:

{ "experimentId": "terminal-bench-2.1", "cells": 12, "incomplete": 0 }

This machine-readable output enables easy integration with CI/CD pipelines and downstream analysis tools.

Concurrency Control and State Safety

To prevent data corruption from concurrent executions, Maka implements exclusive write access through packages/eval/src/attempt-store.ts. Before starting an experiment, the framework writes a temporary "attempt store" to disk.

This mechanism ensures that only one writer can be active for a given experiment directory at any time. If a second process attempts to run the same experiment while it is already executing, the attempt store blocks the new execution, preventing race conditions and state corruption.

Runtime Integration and Experimental Features

The evaluation harness communicates with various runtime hosts (such as packages/runtime-host/src/protocol/web-search.ts) through well-defined contracts. The framework supports toggling experimental features without code changes by using headers like experimental_disabled or OpenAI-Beta, allowing gradual rollout of new capabilities while maintaining stability in production evaluations.

Summary

• Experiments and evaluations in Maka are defined by JSON specifications (experiment.json) stored in the experiments/ directory and validated against the maka.eval.v1 schema.
• The framework enforces strict consistency between on-disk specifications and in-memory representations through packages/eval/src/experiment-directory.ts.
• Execution is orchestrated by packages/eval/src/runner.ts building cells and packages/eval/src/harness-executor.ts delegating to runtimes like Docker or Windows sandboxes.
• Results are aggregated in packages/eval/src/result.ts and emitted as JSON by the CLI for pipeline integration.
• Concurrency protection via packages/eval/src/attempt-store.ts prevents simultaneous writes and data corruption.
• Runtime hosts support feature flags through HTTP headers, enabling safe testing of experimental capabilities.

Frequently Asked Questions

What is the structure of a Maka experiment specification?

A Maka experiment specification is a JSON file named experiment.json that declares a schemaVersion (set to maka.eval.v1), a unique experiment id, an array of subjects representing the code under test, and an array of tasks defining the operations to perform. The packages/eval/src/experiment.ts module validates this structure, ensuring at least one subject and one task exist before execution proceeds.

How does Maka prevent concurrent experiment runs?

Maka prevents concurrent runs through the attempt store mechanism implemented in packages/eval/src/attempt-store.ts. Before execution begins, the framework writes a temporary attempt store to the experiment directory. This acts as a lock file, ensuring only one process can write to the directory at a time. Subsequent attempts to run the same experiment will fail fast rather than corrupt shared state.

What runtimes does Maka support for evaluations?

According to the source code in packages/eval/src/harness-executor.ts, Maka supports multiple runtime environments including Docker containers, Windows sandboxes, and remote hosts. The harness executor abstracts these environments, allowing the same experiment specification to run across different isolation mechanisms without modification.

How are experimental features handled in the evaluation framework?

Experimental features are managed through HTTP headers such as experimental_disabled or OpenAI-Beta in runtime host protocols (as seen in packages/runtime-host/src/protocol/web-search.ts). This header-based approach allows operators to toggle experimental capabilities on or off without deploying new code, providing a safe mechanism for testing new evaluation methods while maintaining production stability.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}