How Apache Maka Manages Immutable Experiment Cell Attempts and Selects the Earliest Valid Result

Maka's evaluation subsystem stores every cell attempt in a write-once, sequence-numbered file layout and deterministically selects the first non-replaceable attempt as the definitive result.

Apache Maka's packages/eval module implements a robust evaluation framework that guarantees data immutability through sequential file storage and deterministic result selection. The system treats each experiment cell execution as an immutable attempt, ensuring that historical evaluation data remains tamper-proof while providing clear semantics for identifying the earliest valid result among potentially multiple runs.

Immutable Attempt Identity and Storage

Sequence-Based File Naming

Each attempt is written to a file whose name encodes a monotonically increasing sequence number (e.g., 000001.json). This zero-padded naming convention creates a natural lexicographic ordering that prevents collisions and ensures temporal consistency across filesystem operations.

Identity Verification

When reading attempts, Maka decodes each file and verifies that the internal cellId and sequence fields match the immutable identity encoded in the filename. If either value mismatches, the system throws an explicit runtime error: "attempt file does not match its immutable identity". This strict validation prevents accidental corruption of historical data and ensures that sequence numbers remain authoritative. This logic is implemented in packages/eval/src/attempt-store.ts (lines 64-67).

Append-Only Storage Guarantees

Atomic Write Operations

The FileAttemptStore class implements an append-only storage mechanism that eliminates race conditions and partial writes. When storing a new attempt, the system first reads the existing attempt list to compute the expected next sequence (expected = last.sequence + 1). The attempt data is then written to a temporary file, fsync-ed to ensure durability, and hard-linked to its final sequence-based name. Only after the hard-link succeeds is the temporary file removed, guaranteeing that a file is never overwritten or reordered. This atomic write pattern appears in packages/eval/src/attempt-store.ts (lines 74-90).

Monotonic Sequence Enforcement

By computing the expected sequence based on the last stored attempt, the system prevents out-of-order writes and ensures that every attempt receives a unique, sequential identifier. This append-only discipline makes the storage layout inherently immutable once written.

Listing and Validating Attempts

Lexicographic Ordering with Zero-Padding

The list(cellId) method reads the cell's directory, filters files matching the nnnnnn.json pattern, and sorts them lexicographically. Because sequence numbers are zero-padded to six digits, simple string sorting produces the correct numeric chronological order. Each file is decoded and validated against its immutable identity before being returned to the caller.

Immutable Ordered Arrays

The validation process ensures that the returned array is strictly ordered by sequence and that each attempt's content matches its filename-encoded identity. This implementation resides in packages/eval/src/attempt-store.ts (lines 49-71).

Deterministic Result Selection

Identifying Replaceable Attempts

Not all attempts are considered final results. An attempt whose status is infra_failed or indeterminate is classified as replaceable, meaning it can be superseded by later successful runs. The isReplaceableAttempt predicate in packages/eval/src/result.ts (lines 87-89) encapsulates this logic, allowing the system to distinguish between transient infrastructure failures and definitive outcomes.

Selecting the Earliest Valid Result

The selectCellResult function receives the ordered list of attempts, performs a defensive sort by sequence, and returns the first attempt that is not replaceable. Because the input list is already ordered by sequence, this effectively selects the earliest successful (or otherwise non-replaceable) attempt for that experiment cell. This algorithm guarantees that even if later attempts succeed after earlier infrastructure failures, the system consistently chooses the first valid result. See packages/eval/src/result.ts (lines 91-95).

Practical Implementation Examples

Creating a Store and Appending Attempts

import { FileAttemptStore } from 'packages/eval/src/attempt-store.js';
import { type CellAttempt } from 'packages/eval/src/result.js';

// Initialise a store in a temporary directory
const store = new FileAttemptStore('/tmp/maka-attempts');

// Example attempt data (normally produced by the harness)
const attempt: CellAttempt = {
  cellId: 'cell-abc',
  sequence: 1,               // must be the next sequence
  startedAt: Date.now(),
  completedAt: Date.now() + 123,
  result: {
    score: 0.85,
    usage: null,
    costUsd: null,
    durationMs: 123,
    status: 'completed',
    failureReason: null,
    artifacts: [],
  },
};

// Store the attempt (writes a file `000001.json` under a hash of the cellId)
await store.append(attempt);

Listing Attempts and Selecting Results

import { selectCellResult } from 'packages/eval/src/result.js';

// Retrieve all attempts for a given cell
const attempts = await store.list('cell-abc');

// Choose the earliest attempt that is not replaceable
const chosen = selectCellResult(attempts);
if (chosen) {
  console.log('Chosen attempt sequence:', chosen.sequence);
  console.log('Result status:', chosen.result.status);
} else {
  console.log('No non-replaceable attempts found');
}

Summary

• Immutable storage: Maka enforces immutability by encoding sequence numbers in filenames and validating internal fields against these identifiers, throwing errors for any mismatches.
• Atomic appends: The FileAttemptStore uses temporary files, fsync, and hard-linking to guarantee that attempts are written exactly once without overwriting existing data.
• Deterministic ordering: Zero-padded sequence numbers enable reliable lexicographic sorting, ensuring attempts are always processed in chronological order.
• Replaceable attempt filtering: The system distinguishes between transient failures (infra_failed, indeterminate) and final results, allowing retries without ambiguity.
• Earliest valid selection: selectCellResult consistently returns the first non-replaceable attempt in the sequence, providing deterministic experiment outcomes.

Frequently Asked Questions

What prevents Maka from overwriting an existing attempt file?

The FileAttemptStore.append method in packages/eval/src/attempt-store.ts computes the expected next sequence based on existing files and uses atomic hard-linking from a temporary file. Because the final filename is derived from the sequence number and the system never writes to existing paths, files cannot be overwritten or reordered after creation.

How does Maka handle attempts that fail due to infrastructure issues?

Attempts with status infra_failed or indeterminate are marked as replaceable by the isReplaceableAttempt function in packages/eval/src/result.ts. When selecting results, selectCellResult skips these replaceable attempts and returns the first subsequent non-replaceable attempt, allowing infrastructure retries without invalidating the experiment's deterministic outcome.

Why does Maka use zero-padded sequence numbers in filenames?

Zero-padding (e.g., 000001.json instead of 1.json) ensures that lexicographic string sorting produces the same ordering as numeric sorting. This allows Maka to rely on simple filesystem directory listings and string comparisons rather than parsing filenames as integers, simplifying the sorting logic in packages/eval/src/attempt-store.ts.

What happens if an attempt file's content doesn't match its filename?

When listing attempts, Maka decodes each file and validates that the internal cellId and sequence fields match the values encoded in the filename. If a mismatch is detected, the system throws a runtime error with the message "attempt file does not match its immutable identity", protecting against filesystem corruption or manual tampering.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}