How OpenMAIC Records and Replays Whiteboard Actions for Cross-Turn Consistency

OpenMAIC captures every whiteboard modification as an immutable action object with a monotonic sequence number, stores these actions in an ordered log, and replays them sequentially at each turn boundary to reconstruct an identical visual state for every agent.

The THU-MAIC/OpenMAIC repository implements a deterministic whiteboard synchronization mechanism that ensures agents evaluate the exact same layout during every turn. By treating whiteboard interactions as a stream of replayable commands rather than mutable state snapshots, the system eliminates visual drift caused by concurrent modifications or differing rendering pipelines.

The Action Recording Pipeline

When users or agents modify the whiteboard canvas, the system does not immediately mutate a shared model. Instead, it generates lightweight descriptors that can be deterministically replayed.

Action Descriptor Structure

In eval/whiteboard-layout/types.ts, each interaction is codified as a WhiteboardAction containing three critical fields:

• seq – A monotonically increasing integer that preserves strict temporal ordering
• type – The operation category ('add', 'move', 'delete', or 'draw')
• payload – Minimal data required to reproduce the change (element IDs, coordinates, content strings, or style objects)

This structure decouples the user interface events from the internal state representation, allowing actions to be serialized, transmitted over network boundaries, and replayed on any compatible runtime.

Recording Logic in state-manager.ts

The recordAction function in eval/whiteboard-layout/state-manager.ts handles the append-only logic:

export interface WhiteboardAction {
  seq: number;
  type: 'add' | 'move' | 'delete' | 'draw';
  payload: Record<string, any>;
}

export function recordAction(
  state: WhiteboardState, 
  action: Omit<WhiteboardAction, 'seq'>
): void {
  const seq = state.nextSeq++;
  const fullAction: WhiteboardAction = { seq, ...action };
  state.log.push(fullAction);  // Append to ordered log
}

The function assigns the next available sequence number and pushes the complete descriptor into state.log, an array that serves as the single source of truth for the current turn's modifications.

Persisting Whiteboard State Between Turns

To maintain consistency when transitioning between agent turns, the action log undergoes serialization and hydration cycles. The state-manager.ts module exposes two utilities for this purpose:

export function serializeLog(state: WhiteboardState): string {
  return JSON.stringify(state.log);
}

export function hydrateLog(serialized: string): WhiteboardAction[] {
  return JSON.parse(serialized) as WhiteboardAction[];
}

When an agent completes its turn, serializeLog converts the accumulated action array into a JSON string that can be stored in session state or transmitted via API. The subsequent turn invokes hydrateLog to receive an identical copy of the action history, ensuring the next agent begins with the exact same operation sequence as its predecessor.

Replay Engine for Deterministic Reconstruction

The critical consistency guarantee comes from the replay mechanism, which rebuilds the whiteboard model from scratch using only the action log, eliminating dependency on mutable shared memory.

Strict Sequence Ordering in replayLog

The replayLog function, implemented in eval/whiteboard-layout/scorer.ts, processes actions in sequence order to reconstruct the visual state:

export function replayLog(actions: WhiteboardAction[]): WhiteboardModel {
  const model: WhiteboardModel = { elements: {} };

  actions
    .sort((a, b) => a.seq - b.seq)  // Enforce strict ordering
    .forEach(action => {
      switch (action.type) {
        case 'add':
          model.elements[action.payload.id] = { ...action.payload };
          break;
        case 'move':
          const el = model.elements[action.payload.id];
          if (el) { 
            el.x = action.payload.x; 
            el.y = action.payload.y; 
          }
          break;
        case 'delete':
          delete model.elements[action.payload.id];
          break;
        case 'draw':
          model.elements[action.payload.id].path.push(action.payload.point);
          break;
      }
    });

  return model;
}

By sorting on seq before execution, the engine guarantees that concurrent or out-of-order deliveries do not corrupt the final state. Each handler operates as a pure function, mutating only the local model object without side effects on external systems.

Integration with Scoring Logic

After reconstruction, the resulting WhiteboardModel feeds directly into the evaluation logic within eval/whiteboard-layout/scorer.ts. This ensures that scoring algorithms analyze the precise visual layout that human users observe, because both views derive from the identical action log processed through the same deterministic replay pipeline.

Summary

• Immutable Actions: Every canvas interaction generates an immutable descriptor containing a sequence number, operation type, and payload.
• Ordered Logging: The recordAction function in eval/whiteboard-layout/state-manager.ts maintains an append-only log that preserves strict temporal ordering via the seq field.
• JSON Serialization: Action logs serialize to JSON for persistence across turn boundaries using serializeLog and hydrateLog.
• Deterministic Replay: The replayLog function reconstructs the whiteboard model from scratch by processing actions in sequence order, ensuring identical state across all agent instances.
• Pure Functional Design: The entire pipeline avoids side effects by rebuilding state rather than mutating shared references, preventing drift between user and agent views.

Frequently Asked Questions

What data structure does OpenMAIC use to store whiteboard actions?

OpenMAIC stores whiteboard actions as an array of WhiteboardAction objects, where each object implements the interface defined in eval/whiteboard-layout/types.ts. The structure includes a seq number for ordering, a type string categorizing the operation, and a payload object containing operation-specific data. This array lives within the WhiteboardState object managed by eval/whiteboard-layout/state-manager.ts.

How does OpenMAIC ensure actions are replayed in the correct order?

The system enforces ordering through the monotonic seq property assigned during recording. When replaying, the replayLog function in eval/whiteboard-layout/scorer.ts explicitly sorts the action array using (a, b) => a.seq - b.seq before processing, ensuring that moves, deletions, and additions execute in the exact sequence they originally occurred.

Why does OpenMAIC rebuild the whiteboard model from scratch each turn?

OpenMAIC employs pure functional reconstruction rather than incremental updates to eliminate state drift. By passing the action log through replayLog to generate a fresh WhiteboardModel at every turn boundary, the system guarantees that all agents—regardless of when they joined the session or how many UI events occurred—compute identical internal representations from the same canonical history.

Where is the whiteboard action log maintained in the codebase?

The action log is maintained primarily in eval/whiteboard-layout/state-manager.ts, which provides the recordAction, serializeLog, and hydrateLog functions. The replay and scoring logic resides in eval/whiteboard-layout/scorer.ts, while the TypeScript interfaces defining the action shapes are declared in eval/whiteboard-layout/types.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}