# What Is the Eval Package in Apache Maka? Understanding Experiment Execution

> Discover the Maka eval package. Learn how it manages experiment specifications and enables verifiable benchmark runs through its execution framework.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: deep-dive
- Published: 2026-09-10

---

**The `@maka/eval` package defines the core semantics of evaluation in the Maka platform, managing experiment specifications, expanding them into runnable cells, and providing the pluggable execution framework that enables verifiable benchmark runs.**

The **eval package** in the Apache Maka repository contains the essential infrastructure for defining, validating, and executing evaluation experiments. Unlike the main Maka runtime, this package does not execute Maka code directly; instead, it establishes the contract between experiment definitions and their execution environments. Located under `packages/eval/src`, the package implements the data models, expansion logic, and security protocols required for reproducible benchmarking across diverse computing substrates.

## Core Responsibilities of the Eval Package

The `@maka/eval` package separates experiment definition from execution through six primary responsibilities.

### Experiment Semantics and Validation

At the heart of the package lies the **experiment specification model** defined in [`src/experiment.ts`](https://github.com/apache/maka/blob/main/src/experiment.ts). The `ExperimentSpec` interface describes the complete structure of an evaluation, including the benchmark, executor configuration, subject definitions, tasks, repetition counts, budget constraints, and verifier settings.

Validation occurs through `parseExperimentSpec` in [`src/spec.ts`](https://github.com/apache/maka/blob/main/src/spec.ts), which performs schema validation and normalization on incoming JSON configurations. This ensures that every experiment adheres to the expected structure before execution proceeds.

### Cartesian-Product Expansion

Once validated, experiments undergo expansion into discrete execution units. The `expandExperiment` function (also in [`src/experiment.ts`](https://github.com/apache/maka/blob/main/src/experiment.ts)) generates every possible **cell**—the unique combination of `task × repetition × subject`—that requires execution. This Cartesian-product approach ensures comprehensive coverage of the experimental design while maintaining clear traceability between specification parameters and individual runs.

### Pluggable Execution Model

The package defines two critical interfaces in [`src/runner.ts`](https://github.com/apache/maka/blob/main/src/runner.ts) that decouple experiment logic from execution mechanics:

- **`ExperimentExecutor`** – Determines how each cell is attempted. Built-in implementations include the Harbor and Pier executors, though custom executors can replace these through dependency injection.
- **`SubjectAdapter`** – Defines how subjects within cells are invoked, supporting both Maka-native subjects and external command-line interfaces.

These interfaces enable Maka to run experiments on local containers, remote clusters, or specialized hardware without modifying the core evaluation logic.

### CLI Integration

The `maka eval` sub-command, implemented in [`packages/cli/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/cli/src/cli.ts), provides the primary entry point for evaluation workflows. This CLI wires default executors and subject adapters together, validates environment prerequisites, and launches the evaluation harness. The `runMakaEvalCli` function orchestrates the dependency injection required to connect the eval package's abstract interfaces to concrete implementations.

### Relay Protocol for Structured Results

To bridge unstructured process outputs with structured result reporting, the package implements a relay protocol in [`src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/src/relay-result-frame.ts). The `writeRelayResult` function emits a single line beginning with `MAKA-EVAL-RESULT-V1` containing the evaluation outcome. This result frame includes authentication via a one-time token, ensuring result integrity even when Docker containers or subprocesses interleave logging output with the actual result data.

### Security and Egress Controls

The evaluation framework enforces strict security boundaries through egress proxying and metering checkpoints. Located within various `egress-proxy` and `metering-checkpoint` modules under `packages/eval/src`, this logic records provider usage, validates audit logs, and prevents unauthorized network access during evaluation runs. These controls guarantee that results reflect genuine computation rather than external data leakage.

## Key Source Files and Architecture

Understanding the eval package requires familiarity with these specific source locations:

- **[`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts)** – Contains `parseExperimentSpec`, the hand-written JSON parser and validator for experiment configurations.
- **[`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts)** – Defines the `ExperimentSpec` type, cell structures, and the `expandExperiment` function for Cartesian-product generation.
- **[`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts)** – Declares the `ExperimentExecutor` and `SubjectAdapter` interfaces that enable pluggable execution.
- **[`packages/eval/src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/relay-result-frame.ts)** – Implements `writeRelayResult` for authenticated, structured result transmission.
- **[`packages/eval/src/harness-executor.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/harness-executor.ts)** – Provides the generic harness implementation used by both Harbor and Pier executors.
- **[`packages/cli/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/cli/src/cli.ts)** – Houses the CLI wiring that invokes the eval package for the `maka eval` sub-command.

## Practical Implementation Examples

### Parsing and Validating Experiment Specifications

To load and validate an experiment definition programmatically:

```typescript
import { readFileSync } from 'node:fs';
import { parseExperimentSpec } from '@maka/eval/src/spec.js';

const json = readFileSync('experiment.json', 'utf8');
const spec = parseExperimentSpec(JSON.parse(json));
console.log('Validated spec ID:', spec.id);

```

This example imports the validator from [`src/spec.ts`](https://github.com/apache/maka/blob/main/src/spec.ts) and ensures the experiment configuration meets all structural requirements before execution.

### Expanding Specs into Execution Cells

After validation, convert the specification into runnable units:

```typescript
import { expandExperiment } from '@maka/eval/src/experiment.js';

const cells = expandExperiment(spec);
console.log('Number of cells to run:', cells.length);
console.log('First cell ID:', cells[0].id);

```

The `expandExperiment` function generates the complete matrix of tasks, subjects, and repetitions defined in the specification.

### Running Evaluations via CLI

Execute a full evaluation from the command line:

```bash
maka eval run experiment.json --out .maka-eval/run-001

```

This command invokes the CLI entry point in [`packages/cli/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/cli/src/cli.ts), which delegates to `runMakaEvalCli` and automatically wires the default Harbor executor and subject adapters.

### Implementing Custom Executors

To replace the built-in execution logic, implement the `ExperimentExecutor` interface:

```typescript
import type { ExperimentExecutor, ExperimentCell } from '@maka/eval/src/experiment.js';

export const myExecutor: ExperimentExecutor = {
  kind: 'my-executor',
  async runAttempt({ cell, signal }, operation) {
    console.log('Running cell', cell.id);
    const result = await operation({ 
      context: {/*...*/}, 
      verify: async () => {/*...*/} 
    });
    return { outcome: 'success', result };
  },
};

```

Register this custom executor through the CLI dependencies (`EvalCliDependencies.loadExecutor`) to override the default execution behavior while retaining the eval package's validation and expansion logic.

## Summary

- The **eval package** defines experiment semantics through `ExperimentSpec` and validates configurations via `parseExperimentSpec` in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts).
- It expands high-level specifications into discrete execution cells using `expandExperiment` from [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts).
- Pluggable execution is enabled by the `ExperimentExecutor` and `SubjectAdapter` interfaces declared in [`packages/eval/src/runner.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/runner.ts).
- The `maka eval` CLI command in [`packages/cli/src/cli.ts`](https://github.com/apache/maka/blob/main/packages/cli/src/cli.ts) integrates the package into the broader Maka workflow.
- Result integrity is guaranteed by the relay protocol (`MAKA-EVAL-RESULT-V1`) implemented in [`packages/eval/src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/relay-result-frame.ts).
- Security controls including egress proxying and metering checkpoints ensure verifiable, tamper-resistant evaluations.

## Frequently Asked Questions

### What distinguishes the eval package from the main Maka runtime?

The eval package does not execute Maka code or create runtime objects directly. Instead, it describes experiments, validates specifications, and provides the plumbing that allows external executors to run cells while guaranteeing trustworthy result reporting. This separation enables Maka to benchmark arbitrary systems, not just native Maka programs.

### How does the eval package ensure result integrity during execution?

The package implements a relay protocol via `writeRelayResult` in [`src/relay-result-frame.ts`](https://github.com/apache/maka/blob/main/src/relay-result-frame.ts), which emits a single structured line (`MAKA-EVAL-RESULT-V1`) containing the evaluation outcome. This frame includes a one-time authentication token that validates the result's origin, preventing tampering even when execution environments produce interleaved or unstructured log output.

### Can I implement custom executors for specific hardware environments?

Yes. The `ExperimentExecutor` interface in [`src/runner.ts`](https://github.com/apache/maka/blob/main/src/runner.ts) allows complete customization of how cells are attempted. By implementing this interface and registering your executor through `EvalCliDependencies.loadExecutor`, you can target specialized hardware, custom container orchestrators, or remote clusters while maintaining compatibility with Maka's experiment specification format and validation logic.

### Where are experiment budget and repetition constraints defined?

Budget constraints, repetition counts, and verifier settings are defined within the `ExperimentSpec` interface located in [`packages/eval/src/experiment.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/experiment.ts). The `parseExperimentSpec` function in [`packages/eval/src/spec.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/spec.ts) validates these parameters during the initial configuration phase, ensuring that resource limits and experimental designs are properly structured before expansion and execution occur.