ActionEngine Unified Execution Layer in OpenMAIC: Technical Deep Dive

The ActionEngine provides a single, unified execution layer in OpenMAIC that routes all 28+ agent-driven actions through one coherent entry point, replacing ad-hoc Vercel AI SDK tools with a shared runtime for both live streaming and offline playback.

The THU-MAIC/OpenMAIC repository implements a centralized unified execution layer to coordinate complex classroom interactions. The ActionEngine consolidates disparate action types—ranging from speech synthesis to whiteboard drawing—into a single orchestration system that supports deterministic replay and real-time execution without code duplication.

Core Architecture of the Unified Execution Layer

The ActionEngine resides in lib/action/engine.ts and serves as the exclusive gateway for all interactive behaviors. According to the repository's architecture documentation, this component executes 28+ action types including speech, whiteboard draw/text/shape/chart, spotlight, and laser effects.

Single Entry Point for All Actions

Every action flows through the execute method defined at lines 14-33 of lib/action/engine.ts. This method accepts an Action union type and routes it to the appropriate handler. By centralizing dispatch logic, the engine ensures consistent error handling, logging, and state management across both live classroom sessions and recorded playbacks.

// lib/action/engine.ts - The core dispatch method
public async execute(action: Action, options?: ExecuteOptions): Promise<void> {
  // Routes to executeSpeech, executeSpotlight, executeWbDraw, etc.
  // based on action.type
}

Dual Execution Modes

The unified execution layer supports two distinct concurrency models:

• Fire-and-forget actions (spotlight, laser) dispatch instantly and return without awaiting completion. These are non-blocking to maintain UI responsiveness during visual effects.
• Synchronous actions (speech, whiteboard drawing, video playback) return a Promise that resolves only after the action fully completes, ensuring sequential dependency chains in lesson scripts.

Key Capabilities of the ActionEngine

Automatic Whiteboard Handling

When whiteboard-related actions invoke while the board is closed, the engine automatically opens it first. The ensureWhiteboardOpen helper (lines 27-30 of lib/action/engine.ts) checks the current whiteboard state and triggers an open animation before executing the draw command, preventing lost strokes or missed LaTeX renders.

Unified Video Resolution

The resolveActionVideoMedia function (lines 31-55) abstracts the lookup of correct video media tasks for any given slide element. This resolver handles deferred restores and playback readiness checks, ensuring that video actions reference the correct media blobs regardless of when the video element registered with the DOM.

Visual Effect Scheduling

Visual effects like spotlight and laser auto-clear after a configurable delay to prevent UI clutter. The scheduleEffectClear mechanism (referenced at lines 7-10 and 18-22) manages timers that automatically remove transient highlights, maintaining a clean presentation state without manual cleanup code in individual action handlers.

Widget Communication

The engine forwards messages to embedded widget iframes via the widgetMessageCallback (lines 70-84). This enables bidirectional communication for actions such as widget_highlight and widget_setState, allowing external React components or third-party embeds to respond to lesson progression events through a standardized message bus.

Playback-Aware State Reset

The resetPlaybackVisualState method (lines 96-105) clears visual effects, pauses active videos, and resets the whiteboard when playback resumes. This ensures deterministic replay by providing a clean slate between runs, preventing state leakage from previous playback iterations into the current session.

Implementation Example

The following pattern initializes the ActionEngine within a React context and demonstrates both execution modes:

import { ActionEngine } from '@/lib/action/engine';
import { useStageStore } from '@/lib/store/stage';
import { createAudioPlayer } from '@/lib/utils/audio-player';

// Initialise the engine once (e.g. in a React context)
const stageStore = useStageStore();
const audioPlayer = createAudioPlayer();
const actionEngine = new ActionEngine(stageStore, audioPlayer);

// 1️⃣ Fire‑and‑forget: spotlight a slide element
await actionEngine.execute({
  type: 'spotlight',
  elementId: 'elem-123',
  dimOpacity: 0.6,
}); // returns immediately

// 2️⃣ Synchronous: speak a sentence (awaits audio end)
await actionEngine.execute({
  type: 'speech',
  audioId: 'audio-456',
}); // resolves when audio finishes

// 3️⃣ Whiteboard drawing: add LaTeX text (auto‑opens board if needed)
await actionEngine.execute({
  type: 'wb_draw_latex',
  elementId: 'wb-1',
  content: '$$E=mc^2$$',
});

// 4️⃣ Video playback with abort support
const abort = new AbortController();
await actionEngine.execute(
  { type: 'play_video', elementId: 'vid-78' },
  { signal: abort.signal }
);

Integration Points

The unified execution layer integrates with two primary runtime environments:

• Live chat runtime: components/chat/use-chat-sessions.ts instantiates the engine to process streaming AI agent outputs in real time.
• Playback state manager: eval/whiteboard-layout/state-manager.ts utilizes the same engine to drive deterministic replay, ensuring that recorded lessons execute through identical code paths as live sessions.

This architectural symmetry eliminates the need for separate playback and live toolchains, reducing maintenance overhead and guaranteeing behavioral consistency across modes.

Summary

• The ActionEngine in lib/action/engine.ts provides the unified execution layer for all OpenMAIC interactions.
• The execute method serves as the single entry point for 28+ action types, implementing both fire-and-forget and synchronous execution semantics.
• Automatic whiteboard handling via ensureWhiteboardOpen prevents lost content when the board is closed.
• Video resolution and effect scheduling are abstracted into reusable helpers that maintain clean UI states.
• The engine supports widget iframe communication and playback state resets, enabling seamless integration with both live streaming and recorded playback systems.

Frequently Asked Questions

What file contains the ActionEngine implementation?

The core implementation resides in lib/action/engine.ts within the THU-MAIC/OpenMAIC repository. This file defines the ActionEngine class, the execute method, and all supporting utilities including resolveActionVideoMedia, ensureWhiteboardOpen, and resetPlaybackVisualState.

How does the ActionEngine handle different action types?

The engine uses a discriminated union Action type to route operations through a single execute method. Internally, this method switches on the type property and delegates to specialized handlers like executeSpeech, executeSpotlight, or executeWbDraw. This pattern allows new action types to be added by extending the union type and implementing a corresponding execute* method without modifying the surrounding runtime infrastructure.

What is the difference between fire-and-forget and synchronous actions?

Fire-and-forget actions (such as spotlight or laser effects) dispatch immediately and return control to the caller without waiting for completion. Synchronous actions (such as speech or video playback) return a Promise that resolves only after the action fully completes. This distinction allows lesson scripts to sequence dependent operations—such as waiting for audio to finish before advancing slides—while keeping transient visual effects non-blocking.

How does the engine support both live and playback modes?

The ActionEngine provides a unified execution layer that both online streaming and offline playback paths share. The same execute method drives live classroom interactions in components/chat/use-chat-sessions.ts and deterministic replay in eval/whiteboard-layout/state-manager.ts. By using identical code paths, OpenMAIC guarantees that recorded lessons behave exactly as they did during the live session, with resetPlaybackVisualState ensuring clean state initialization between playback runs.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}