# ActionEngine Unified Execution Layer in OpenMAIC: Technical Deep Dive

> Explore the ActionEngine unified execution layer in OpenMAIC. Discover how it streamlines agent actions for live streaming and offline playback, offering a single entry point for enhanced efficiency.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: deep-dive
- Published: 2026-09-05

---

**The ActionEngine provides a single, unified execution layer in OpenMAIC that routes all 28+ agent-driven actions through one coherent entry point, replacing ad-hoc Vercel AI SDK tools with a shared runtime for both live streaming and offline playback.**

The THU-MAIC/OpenMAIC repository implements a centralized **unified execution layer** to coordinate complex classroom interactions. The ActionEngine consolidates disparate action types—ranging from speech synthesis to whiteboard drawing—into a single orchestration system that supports deterministic replay and real-time execution without code duplication.

## Core Architecture of the Unified Execution Layer

The ActionEngine resides in [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts) and serves as the exclusive gateway for all interactive behaviors. According to the repository's architecture documentation, this component executes **28+ action types** including speech, whiteboard draw/text/shape/chart, spotlight, and laser effects.

### Single Entry Point for All Actions

Every action flows through the `execute` method defined at lines 14-33 of [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts). This method accepts an `Action` union type and routes it to the appropriate handler. By centralizing dispatch logic, the engine ensures consistent error handling, logging, and state management across both live classroom sessions and recorded playbacks.

```typescript
// lib/action/engine.ts - The core dispatch method
public async execute(action: Action, options?: ExecuteOptions): Promise<void> {
  // Routes to executeSpeech, executeSpotlight, executeWbDraw, etc.
  // based on action.type
}

```

### Dual Execution Modes

The unified execution layer supports two distinct concurrency models:

- **Fire-and-forget actions** (spotlight, laser) dispatch instantly and return without awaiting completion. These are non-blocking to maintain UI responsiveness during visual effects.
- **Synchronous actions** (speech, whiteboard drawing, video playback) return a `Promise` that resolves only after the action fully completes, ensuring sequential dependency chains in lesson scripts.

## Key Capabilities of the ActionEngine

### Automatic Whiteboard Handling

When whiteboard-related actions invoke while the board is closed, the engine automatically opens it first. The `ensureWhiteboardOpen` helper (lines 27-30 of [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts)) checks the current whiteboard state and triggers an open animation before executing the draw command, preventing lost strokes or missed LaTeX renders.

### Unified Video Resolution

The `resolveActionVideoMedia` function (lines 31-55) abstracts the lookup of correct video media tasks for any given slide element. This resolver handles deferred restores and playback readiness checks, ensuring that video actions reference the correct media blobs regardless of when the video element registered with the DOM.

### Visual Effect Scheduling

Visual effects like spotlight and laser auto-clear after a configurable delay to prevent UI clutter. The `scheduleEffectClear` mechanism (referenced at lines 7-10 and 18-22) manages timers that automatically remove transient highlights, maintaining a clean presentation state without manual cleanup code in individual action handlers.

### Widget Communication

The engine forwards messages to embedded widget iframes via the `widgetMessageCallback` (lines 70-84). This enables bidirectional communication for actions such as `widget_highlight` and `widget_setState`, allowing external React components or third-party embeds to respond to lesson progression events through a standardized message bus.

### Playback-Aware State Reset

The `resetPlaybackVisualState` method (lines 96-105) clears visual effects, pauses active videos, and resets the whiteboard when playback resumes. This ensures deterministic replay by providing a clean slate between runs, preventing state leakage from previous playback iterations into the current session.

## Implementation Example

The following pattern initializes the ActionEngine within a React context and demonstrates both execution modes:

```typescript
import { ActionEngine } from '@/lib/action/engine';
import { useStageStore } from '@/lib/store/stage';
import { createAudioPlayer } from '@/lib/utils/audio-player';

// Initialise the engine once (e.g. in a React context)
const stageStore = useStageStore();
const audioPlayer = createAudioPlayer();
const actionEngine = new ActionEngine(stageStore, audioPlayer);

// 1️⃣ Fire‑and‑forget: spotlight a slide element
await actionEngine.execute({
  type: 'spotlight',
  elementId: 'elem-123',
  dimOpacity: 0.6,
}); // returns immediately

// 2️⃣ Synchronous: speak a sentence (awaits audio end)
await actionEngine.execute({
  type: 'speech',
  audioId: 'audio-456',
}); // resolves when audio finishes

// 3️⃣ Whiteboard drawing: add LaTeX text (auto‑opens board if needed)
await actionEngine.execute({
  type: 'wb_draw_latex',
  elementId: 'wb-1',
  content: '$$E=mc^2$$',
});

// 4️⃣ Video playback with abort support
const abort = new AbortController();
await actionEngine.execute(
  { type: 'play_video', elementId: 'vid-78' },
  { signal: abort.signal }
);

```

## Integration Points

The unified execution layer integrates with two primary runtime environments:

- **Live chat runtime**: [`components/chat/use-chat-sessions.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/components/chat/use-chat-sessions.ts) instantiates the engine to process streaming AI agent outputs in real time.
- **Playback state manager**: [`eval/whiteboard-layout/state-manager.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/eval/whiteboard-layout/state-manager.ts) utilizes the same engine to drive deterministic replay, ensuring that recorded lessons execute through identical code paths as live sessions.

This architectural symmetry eliminates the need for separate playback and live toolchains, reducing maintenance overhead and guaranteeing behavioral consistency across modes.

## Summary

- The **ActionEngine** in [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts) provides the unified execution layer for all OpenMAIC interactions.
- The `execute` method serves as the single entry point for 28+ action types, implementing both fire-and-forget and synchronous execution semantics.
- **Automatic whiteboard handling** via `ensureWhiteboardOpen` prevents lost content when the board is closed.
- **Video resolution** and **effect scheduling** are abstracted into reusable helpers that maintain clean UI states.
- The engine supports **widget iframe communication** and **playback state resets**, enabling seamless integration with both live streaming and recorded playback systems.

## Frequently Asked Questions

### What file contains the ActionEngine implementation?

The core implementation resides in [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts) within the THU-MAIC/OpenMAIC repository. This file defines the `ActionEngine` class, the `execute` method, and all supporting utilities including `resolveActionVideoMedia`, `ensureWhiteboardOpen`, and `resetPlaybackVisualState`.

### How does the ActionEngine handle different action types?

The engine uses a discriminated union `Action` type to route operations through a single `execute` method. Internally, this method switches on the `type` property and delegates to specialized handlers like `executeSpeech`, `executeSpotlight`, or `executeWbDraw`. This pattern allows new action types to be added by extending the union type and implementing a corresponding `execute*` method without modifying the surrounding runtime infrastructure.

### What is the difference between fire-and-forget and synchronous actions?

**Fire-and-forget** actions (such as spotlight or laser effects) dispatch immediately and return control to the caller without waiting for completion. **Synchronous** actions (such as speech or video playback) return a `Promise` that resolves only after the action fully completes. This distinction allows lesson scripts to sequence dependent operations—such as waiting for audio to finish before advancing slides—while keeping transient visual effects non-blocking.

### How does the engine support both live and playback modes?

The ActionEngine provides a **unified execution layer** that both online streaming and offline playback paths share. The same `execute` method drives live classroom interactions in [`components/chat/use-chat-sessions.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/components/chat/use-chat-sessions.ts) and deterministic replay in [`eval/whiteboard-layout/state-manager.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/eval/whiteboard-layout/state-manager.ts). By using identical code paths, OpenMAIC guarantees that recorded lessons behave exactly as they did during the live session, with `resetPlaybackVisualState` ensuring clean state initialization between playback runs.