# How Apache Maka Manages Immutable Experiment Cell Attempts and Selects the Earliest Valid Result

> Discover how Apache Maka's evaluation subsystem manages immutable experiment cells using sequence-numbered files and selects the earliest valid result for definitive outcomes.

- Repository: [The Apache Software Foundation/maka](https://github.com/apache/maka)
- Tags: internals
- Published: 2026-09-01

---

**Maka's evaluation subsystem stores every cell attempt in a write-once, sequence-numbered file layout and deterministically selects the first non-replaceable attempt as the definitive result.**

Apache Maka's `packages/eval` module implements a robust evaluation framework that guarantees data immutability through sequential file storage and deterministic result selection. The system treats each experiment cell execution as an immutable attempt, ensuring that historical evaluation data remains tamper-proof while providing clear semantics for identifying the earliest valid result among potentially multiple runs.

## Immutable Attempt Identity and Storage

### Sequence-Based File Naming

Each attempt is written to a file whose name encodes a monotonically increasing sequence number (e.g., [`000001.json`](https://github.com/apache/maka/blob/main/000001.json)). This zero-padded naming convention creates a natural lexicographic ordering that prevents collisions and ensures temporal consistency across filesystem operations.

### Identity Verification

When reading attempts, Maka decodes each file and verifies that the internal `cellId` and `sequence` fields match the immutable identity encoded in the filename. If either value mismatches, the system throws an explicit runtime error: "attempt file does not match its immutable identity". This strict validation prevents accidental corruption of historical data and ensures that sequence numbers remain authoritative. This logic is implemented in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts) (lines 64-67).

## Append-Only Storage Guarantees

### Atomic Write Operations

The `FileAttemptStore` class implements an append-only storage mechanism that eliminates race conditions and partial writes. When storing a new attempt, the system first reads the existing attempt list to compute the expected next sequence (`expected = last.sequence + 1`). The attempt data is then written to a temporary file, `fsync`-ed to ensure durability, and hard-linked to its final sequence-based name. Only after the hard-link succeeds is the temporary file removed, guaranteeing that a file is never overwritten or reordered. This atomic write pattern appears in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts) (lines 74-90).

### Monotonic Sequence Enforcement

By computing the expected sequence based on the last stored attempt, the system prevents out-of-order writes and ensures that every attempt receives a unique, sequential identifier. This append-only discipline makes the storage layout inherently immutable once written.

## Listing and Validating Attempts

### Lexicographic Ordering with Zero-Padding

The `list(cellId)` method reads the cell's directory, filters files matching the [`nnnnnn.json`](https://github.com/apache/maka/blob/main/nnnnnn.json) pattern, and sorts them lexicographically. Because sequence numbers are zero-padded to six digits, simple string sorting produces the correct numeric chronological order. Each file is decoded and validated against its immutable identity before being returned to the caller.

### Immutable Ordered Arrays

The validation process ensures that the returned array is strictly ordered by sequence and that each attempt's content matches its filename-encoded identity. This implementation resides in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts) (lines 49-71).

## Deterministic Result Selection

### Identifying Replaceable Attempts

Not all attempts are considered final results. An attempt whose status is `infra_failed` or `indeterminate` is classified as *replaceable*, meaning it can be superseded by later successful runs. The `isReplaceableAttempt` predicate in [`packages/eval/src/result.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/result.ts) (lines 87-89) encapsulates this logic, allowing the system to distinguish between transient infrastructure failures and definitive outcomes.

### Selecting the Earliest Valid Result

The `selectCellResult` function receives the ordered list of attempts, performs a defensive sort by `sequence`, and returns the **first** attempt that is not replaceable. Because the input list is already ordered by sequence, this effectively selects the earliest successful (or otherwise non-replaceable) attempt for that experiment cell. This algorithm guarantees that even if later attempts succeed after earlier infrastructure failures, the system consistently chooses the first valid result. See [`packages/eval/src/result.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/result.ts) (lines 91-95).

## Practical Implementation Examples

### Creating a Store and Appending Attempts

```typescript
import { FileAttemptStore } from 'packages/eval/src/attempt-store.js';
import { type CellAttempt } from 'packages/eval/src/result.js';

// Initialise a store in a temporary directory
const store = new FileAttemptStore('/tmp/maka-attempts');

// Example attempt data (normally produced by the harness)
const attempt: CellAttempt = {
  cellId: 'cell-abc',
  sequence: 1,               // must be the next sequence
  startedAt: Date.now(),
  completedAt: Date.now() + 123,
  result: {
    score: 0.85,
    usage: null,
    costUsd: null,
    durationMs: 123,
    status: 'completed',
    failureReason: null,
    artifacts: [],
  },
};

// Store the attempt (writes a file `000001.json` under a hash of the cellId)
await store.append(attempt);

```

### Listing Attempts and Selecting Results

```typescript
import { selectCellResult } from 'packages/eval/src/result.js';

// Retrieve all attempts for a given cell
const attempts = await store.list('cell-abc');

// Choose the earliest attempt that is not replaceable
const chosen = selectCellResult(attempts);
if (chosen) {
  console.log('Chosen attempt sequence:', chosen.sequence);
  console.log('Result status:', chosen.result.status);
} else {
  console.log('No non-replaceable attempts found');
}

```

## Summary

- **Immutable storage**: Maka enforces immutability by encoding sequence numbers in filenames and validating internal fields against these identifiers, throwing errors for any mismatches.
- **Atomic appends**: The `FileAttemptStore` uses temporary files, `fsync`, and hard-linking to guarantee that attempts are written exactly once without overwriting existing data.
- **Deterministic ordering**: Zero-padded sequence numbers enable reliable lexicographic sorting, ensuring attempts are always processed in chronological order.
- **Replaceable attempt filtering**: The system distinguishes between transient failures (`infra_failed`, `indeterminate`) and final results, allowing retries without ambiguity.
- **Earliest valid selection**: `selectCellResult` consistently returns the first non-replaceable attempt in the sequence, providing deterministic experiment outcomes.

## Frequently Asked Questions

### What prevents Maka from overwriting an existing attempt file?

The `FileAttemptStore.append` method in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts) computes the expected next sequence based on existing files and uses atomic hard-linking from a temporary file. Because the final filename is derived from the sequence number and the system never writes to existing paths, files cannot be overwritten or reordered after creation.

### How does Maka handle attempts that fail due to infrastructure issues?

Attempts with status `infra_failed` or `indeterminate` are marked as replaceable by the `isReplaceableAttempt` function in [`packages/eval/src/result.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/result.ts). When selecting results, `selectCellResult` skips these replaceable attempts and returns the first subsequent non-replaceable attempt, allowing infrastructure retries without invalidating the experiment's deterministic outcome.

### Why does Maka use zero-padded sequence numbers in filenames?

Zero-padding (e.g., [`000001.json`](https://github.com/apache/maka/blob/main/000001.json) instead of [`1.json`](https://github.com/apache/maka/blob/main/1.json)) ensures that lexicographic string sorting produces the same ordering as numeric sorting. This allows Maka to rely on simple filesystem directory listings and string comparisons rather than parsing filenames as integers, simplifying the sorting logic in [`packages/eval/src/attempt-store.ts`](https://github.com/apache/maka/blob/main/packages/eval/src/attempt-store.ts).

### What happens if an attempt file's content doesn't match its filename?

When listing attempts, Maka decodes each file and validates that the internal `cellId` and `sequence` fields match the values encoded in the filename. If a mismatch is detected, the system throws a runtime error with the message "attempt file does not match its immutable identity", protecting against filesystem corruption or manual tampering.