How ActionEngine Consumes Scene.actions in the OpenMAIC Playback Engine

The Playback Engine consumes Scene.actions by iterating over scene arrays and delegating each action to the ActionEngine through a resolution cursor that handles both fire-and-forget and synchronous awaitable execution patterns.

The OpenMAIC lecture playback system orchestrates multimedia presentations through a decoupled architecture where the Playback Engine manages timeline state while the ActionEngine handles concrete execution. Understanding how these components interact with Scene.actions arrays is essential for extending the platform's 28+ supported action types or debugging lecture playback issues in the THU-MAIC/OpenMAIC repository.

Core Architecture and Initialization

The playback system separates concerns between timeline management and action implementation. The Playback Engine (lib/playback/engine.ts) maintains the playback state machine, while the ActionEngine (lib/action/engine.ts) encapsulates the side effects for every supported action type.

PlaybackEngine Construction

The dependency injection pattern begins in the PlaybackEngine constructor (lines 62‑79 of lib/playback/engine.ts). The engine receives a pre-configured ActionEngine instance that has already been initialized with the StageStore and AudioPlayer dependencies:

import { ActionEngine } from '@/lib/action/engine';
import { PlaybackEngine } from '@/lib/playback/engine';
import { createStageAPI } from '@/lib/api/stage-api';
import { useAudioPlayer } from '@/lib/utils/audio-player';

const stageStore = createStageAPI();
const audioPlayer = useAudioPlayer();
const actionEngine = new ActionEngine(stageStore, audioPlayer);

const playback = new PlaybackEngine(scenes, actionEngine, audioPlayer, callbacks);

This initialization ensures the ActionEngine has access to the canvas, whiteboard history, and media generation stores required for applying visual and audio effects.

Resolving and Delegating Scene.actions

The consumption of Scene.actions follows a deterministic resolution and delegation pattern. Each Scene interface (defined in lib/types/stage.ts) contains an actions: Action[] array that the engine traverses sequentially.

Resolving the Playback Cursor

The processNext method (lines 531‑544) calls resolvePlaybackCursor from lib/choreography/index.ts to determine the next action and its associated scene ID. This resolution handles scene transitions and maintains the actionIndex pointer within the playback state:

• Returns the current action object
• Provides the containing scene identifier
• Manages progression logic for the idle, playing, paused, and live states

Delegating Execution to ActionEngine

Inside the main execution switch statement (lines 842‑894 of lib/playback/engine.ts), the engine routes resolved actions to the ActionEngine's execute method (implemented in lib/action/engine.ts, lines 14‑86). The delegation pattern distinguishes between two execution semantics:

Fire‑and‑forget actions (spotlight, laser) execute immediately without blocking the timeline:

this.actionEngine.execute(action); // lines 862‑864

Synchronous actions (speech, whiteboard, video, widget) await completion before advancing:

await this.actionEngine.execute(action); // lines 835‑842

Action Execution Patterns

The ActionEngine's execute method performs the concrete work for each action type. After resolution, it updates the canvas, whiteboard, media elements, or sends widget messages, then returns control to the Playback Engine.

Fire-and-Forget Actions

Visual pointer actions like spotlight and laser do not block playback progression. The engine invokes these without awaiting, allowing the timeline to continue while visual effects render independently.

Synchronous Awaitable Actions

Content-heavy actions require completion guarantees. Speech synthesis, whiteboard drawing sequences, video playback, and widget interactions return Promises that resolve when the action finishes. The Playback Engine increments this.actionIndex++ only after the ActionEngine resolves, ensuring sequential integrity.

Practical Implementation Examples

Manual Action Execution

You can bypass the playback loop and execute specific actions directly:

const firstAction = scenes[0].actions?.[0];
if (firstAction) {
  await actionEngine.execute(firstAction);
}

Jump Navigation with State Preservation

The engine supports non-linear navigation while preserving whiteboard history:

await playback.jumpToAction(5, { autoplay: true });

This pre-executes preceding whiteboard actions silently to establish visual state, then resumes normal consumption of Scene.actions from index 5.

Summary

• The Playback Engine acts as a stateful cursor over Scene.actions arrays, managing transitions between scenes and playback states.
• Action resolution occurs through resolvePlaybackCursor in lib/choreography/index.ts, which determines the next action and scene context.
• Execution delegation routes each action to ActionEngine.execute with distinct handling for fire‑and‑forget versus synchronous awaitable operations.
• The ActionEngine encapsulates implementation details for 28+ action types, including speech, whiteboard drawing, laser pointers, and widget interactions.
• Consumption patterns support both sequential playback and random access navigation via jumpToAction, with automatic state reconstruction.

Frequently Asked Questions

How does the Playback Engine handle scene transitions when consuming Scene.actions?

The resolvePlaybackCursor function in lib/choreography/index.ts computes the next action across scene boundaries. When the current scene's actions array exhausts, the cursor automatically advances to the next scene's first action while preserving playback continuity and invoking scene change callbacks.

What is the difference between fire-and-forget and synchronous actions in OpenMAIC?

Fire-and-forget actions (spotlight, laser) invoke this.actionEngine.execute(action) without an await statement, allowing immediate cursor advancement in the switch block (lines 862‑864). Synchronous actions (speech, video, whiteboard) use await this.actionEngine.execute(action) (lines 835‑842) to block progression until the action completes, ensuring temporal alignment between audio, visual, and interactive elements.

Can I execute Scene.actions outside of the Playback Engine loop?

Yes. The ActionEngine is designed for independent use. Instantiate it with a StageStore and AudioPlayer, then call await actionEngine.execute(action) on any action object from Scene.actions. This is useful for preview modes, testing individual actions, or implementing custom playback controllers.

Where are the action type definitions located in the OpenMAIC repository?

Type definitions for all 28+ supported actions reside in lib/types/action.ts. This includes interfaces like SpeechAction, SpotlightAction, WhiteboardAction, and WidgetAction. The Scene interface in lib/types/stage.ts declares the actions: Action[] property that the Playback Engine consumes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}