Maka Subject vs External Subject: Understanding the Evaluation Framework in Apache Maka

A Maka subject executes native workloads through the Runtime Host protocol boundary, while an external subject runs arbitrary commands via a generic wrapper that requires toolchain verification.

Apache Maka's evaluation framework (@maka/eval) provides two distinct models for executing workloads inside an experiment cell. Understanding the distinction between a Maka subject and an external subject is essential for configuring benchmarks that range from native Maka tasks to third-party command-line tools.

Core Architectural Differences

The fundamental separation lies in runtime ownership and protocol requirements. Maka subjects operate as first-class citizens within the Maka ecosystem, while external subjects function as sandboxed wrappers for arbitrary executables.

Maka Subjects (Native Execution)

A Maka subject represents a Maka-owned workload that executes exclusively through the public Runtime Host client/protocol boundary. According to ARCHITECTURE.md, these subjects "execute only through Runtime Host" and always cross the public protocol boundary during execution.

Key characteristics include:

• Native Integration: The Runtime Host launches the subject inside a dedicated Host root, preserving session identity, continuation state, and toolchain permissions.
• No Toolchain Verification: Because the workload runs inside the Runtime Host's own environment, external toolchain verification is unnecessary.
• Uniform Result Contract: Returns structured results using the protocol-v1 format that the Evaluation client interprets directly.

External Subjects (Generic Execution)

An external subject provides a flexible mechanism to benchmark arbitrary commands supplied by the user. As documented in ARCHITECTURE.md, this is a "generic external subject adapter" capable of running "any external program, provided the command and arguments are declared."

Key characteristics include:

• Command Wrapper: The Runtime Host launches harbor-external-subject.js, which then spawns the declared user command.
• Toolchain Verification: Before execution, the framework verifies the required toolchain (e.g., Python, Node) exists. The verification logic resides in packages/eval/src/external-subject.ts at lines 45-64.
• Dual Result Contracts: Supports either exit-code (simple success/failure) or protocol-v1 (structured results), though the latter requires the bundled external wrapper.

Implementation Details and Source Code

The adapter implementations in @maka/eval formalize these behavioral differences through distinct factory functions and execution paths.

Adapter Factory Functions

In packages/eval/src/maka-subject.ts, the createMakaSubjectAdapter() function instantiates the native adapter:

import { createMakaSubjectAdapter } from '@maka/eval';

// Creates a native Maka subject (kind = 'maka')
const makaAdapter = createMakaSubjectAdapter();

Conversely, packages/eval/src/external-subject.ts exports createExternalSubjectAdapter():

import { createExternalSubjectAdapter } from '@maka/eval';

// Creates a generic external wrapper (kind = 'external')
const externalAdapter = createExternalSubjectAdapter();

Execution Path and Verification

The Runtime Host handles launch procedures differently for each type. Maka subjects receive dedicated Host root environments with preserved identity contexts. External subjects trigger the external wrapper script, which manages process spawning and toolchain validation.

The toolchain verification in external-subject.ts (lines 45-64) ensures the declared runtime environment exists before attempting execution. This verification step is absent in the Maka subject flow, as those workloads inherit the Runtime Host's verified environment.

Result Contracts and Failure Modes

The two subject types diverge significantly in how they report outcomes and handle errors.

Structured Results vs Exit Codes

Maka subjects exclusively return protocol-v1 structured results containing detailed metrics, continuation state, and cost/usage data.

External subjects offer flexibility through two result contracts:

1. exit-code: Binary success/failure based on process exit status
2. protocol-v1: Full structured results (available only when using the bundled external wrapper)

Error Handling and Artifacts

Failure reasons and diagnostic artifacts differ between the implementations. In packages/eval/src/external-subject.ts, specific timeout errors appear at lines 14-20 ("external subject exceeded the framework timeout"), while execution scope failures are defined at lines 28-36 ("external subject execution scope was unavailable").

Maka subjects generate distinct error messages such as "Maka subject failed …" or "Maka subject exceeded the framework timeout".

The artifact collections also vary:

• Maka subjects: Include toolchain-specific artifacts, continuation state, and detailed metering
• External subjects: Always produce an external_process artifact containing the exit code, plus any recovered metering artifacts (see lines 21-27 and 38-44 in external-subject.ts)

Practical Usage Examples

Both adapters can coexist within the same experiment definition. The following example demonstrates configuring a cell with both subject types using the core runner:

import { runExperiment, createMakaSubjectAdapter, createExternalSubjectAdapter } from '@maka/eval';

await runExperiment({
  // experiment configuration...
  subjects: [
    createMakaSubjectAdapter(),      // Native Maka workload
    createExternalSubjectAdapter(),  // External command wrapper
  ],
});

When defining the external subject, you must specify the command and arguments, whereas the Maka subject derives its execution parameters from the Runtime Host protocol configuration.

Summary

• Maka subjects execute native workloads through the Runtime Host protocol boundary without requiring toolchain verification, returning uniform protocol-v1 structured results.
• External subjects wrap arbitrary user commands, require explicit toolchain verification (lines 45-64 in external-subject.ts), and support both exit-code and protocol-v1 result contracts.
• The external wrapper (harbor-external-subject.js) mediates execution for external subjects, while Maka subjects run directly within dedicated Host roots.
• Failure modes and artifacts are type-specific, with external subjects generating distinct error messages and external_process artifacts.

Frequently Asked Questions

Do external subjects require specific runtime dependencies?

Yes. According to the implementation in packages/eval/src/external-subject.ts (lines 45-64), external subjects require explicit toolchain verification before execution. The framework validates that the declared runtime (e.g., Python, Node.js) is available in the environment, unlike Maka subjects which inherit the Runtime Host's pre-verified environment.

Can external subjects return the same detailed metrics as Maka subjects?

External subjects can return protocol-v1 structured results equivalent to Maka subjects, but only when using the bundled external wrapper (harbor-external-subject.js). If configured for basic exit-code contracts, they return only binary success/failure status without detailed metering or continuation state.

What causes an "external subject execution scope was unavailable" error?

This error originates in packages/eval/src/external-subject.ts (lines 28-36) when the Runtime Host cannot establish the execution context required to spawn the external wrapper. This typically indicates environmental constraints or resource limitations that prevent the external subject adapter from initializing the command sandbox.

How do I configure both subject types in a single experiment?

Import both createMakaSubjectAdapter from packages/eval/src/maka-subject.ts and createExternalSubjectAdapter from packages/eval/src/external-subject.ts, then pass both adapters to the experiment runner's subjects array. The runExperiment function in packages/eval/src/runner.ts orchestrates execution across heterogeneous subject types within the same cell.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}