# How ActionEngine Consumes Scene.actions in the OpenMAIC Playback Engine

> Learn how ActionEngine consumes Scene actions in the OpenMAIC Playback Engine. Discover iteration, delegation, and execution patterns for seamless action management.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: internals
- Published: 2026-08-30

---

**The Playback Engine consumes Scene.actions by iterating over scene arrays and delegating each action to the ActionEngine through a resolution cursor that handles both fire-and-forget and synchronous awaitable execution patterns.**

The OpenMAIC lecture playback system orchestrates multimedia presentations through a decoupled architecture where the **Playback Engine** manages timeline state while the **ActionEngine** handles concrete execution. Understanding how these components interact with `Scene.actions` arrays is essential for extending the platform's 28+ supported action types or debugging lecture playback issues in the THU-MAIC/OpenMAIC repository.

## Core Architecture and Initialization

The playback system separates concerns between timeline management and action implementation. The **Playback Engine** ([`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts)) maintains the playback state machine, while the **ActionEngine** ([`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts)) encapsulates the side effects for every supported action type.

### PlaybackEngine Construction

The dependency injection pattern begins in the `PlaybackEngine` constructor (lines 62‑79 of [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts)). The engine receives a pre-configured `ActionEngine` instance that has already been initialized with the **StageStore** and **AudioPlayer** dependencies:

```typescript
import { ActionEngine } from '@/lib/action/engine';
import { PlaybackEngine } from '@/lib/playback/engine';
import { createStageAPI } from '@/lib/api/stage-api';
import { useAudioPlayer } from '@/lib/utils/audio-player';

const stageStore = createStageAPI();
const audioPlayer = useAudioPlayer();
const actionEngine = new ActionEngine(stageStore, audioPlayer);

const playback = new PlaybackEngine(scenes, actionEngine, audioPlayer, callbacks);

```

This initialization ensures the ActionEngine has access to the canvas, whiteboard history, and media generation stores required for applying visual and audio effects.

## Resolving and Delegating Scene.actions

The consumption of `Scene.actions` follows a deterministic resolution and delegation pattern. Each **Scene** interface (defined in [`lib/types/stage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/stage.ts)) contains an `actions: Action[]` array that the engine traverses sequentially.

### Resolving the Playback Cursor

The `processNext` method (lines 531‑544) calls `resolvePlaybackCursor` from [`lib/choreography/index.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/choreography/index.ts) to determine the next action and its associated scene ID. This resolution handles scene transitions and maintains the `actionIndex` pointer within the playback state:

- Returns the current `action` object
- Provides the containing scene identifier  
- Manages progression logic for the `idle`, `playing`, `paused`, and `live` states

### Delegating Execution to ActionEngine

Inside the main execution switch statement (lines 842‑894 of [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts)), the engine routes resolved actions to the ActionEngine's `execute` method (implemented in [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts), lines 14‑86). The delegation pattern distinguishes between two execution semantics:

**Fire‑and‑forget actions** (`spotlight`, `laser`) execute immediately without blocking the timeline:

```typescript
this.actionEngine.execute(action); // lines 862‑864

```

**Synchronous actions** (speech, whiteboard, video, widget) await completion before advancing:

```typescript
await this.actionEngine.execute(action); // lines 835‑842

```

## Action Execution Patterns

The ActionEngine's `execute` method performs the concrete work for each action type. After resolution, it updates the canvas, whiteboard, media elements, or sends widget messages, then returns control to the Playback Engine.

### Fire-and-Forget Actions

Visual pointer actions like **spotlight** and **laser** do not block playback progression. The engine invokes these without awaiting, allowing the timeline to continue while visual effects render independently.

### Synchronous Awaitable Actions

Content-heavy actions require completion guarantees. Speech synthesis, whiteboard drawing sequences, video playback, and widget interactions return Promises that resolve when the action finishes. The Playback Engine increments `this.actionIndex++` only after the ActionEngine resolves, ensuring sequential integrity.

## Practical Implementation Examples

### Manual Action Execution

You can bypass the playback loop and execute specific actions directly:

```typescript
const firstAction = scenes[0].actions?.[0];
if (firstAction) {
  await actionEngine.execute(firstAction);
}

```

### Jump Navigation with State Preservation

The engine supports non-linear navigation while preserving whiteboard history:

```typescript
await playback.jumpToAction(5, { autoplay: true });

```

This pre-executes preceding whiteboard actions silently to establish visual state, then resumes normal consumption of `Scene.actions` from index 5.

## Summary

- The **Playback Engine** acts as a stateful cursor over `Scene.actions` arrays, managing transitions between scenes and playback states.
- **Action resolution** occurs through `resolvePlaybackCursor` in [`lib/choreography/index.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/choreography/index.ts), which determines the next action and scene context.
- **Execution delegation** routes each action to `ActionEngine.execute` with distinct handling for fire‑and‑forget versus synchronous awaitable operations.
- The **ActionEngine** encapsulates implementation details for 28+ action types, including speech, whiteboard drawing, laser pointers, and widget interactions.
- Consumption patterns support both sequential playback and random access navigation via `jumpToAction`, with automatic state reconstruction.

## Frequently Asked Questions

### How does the Playback Engine handle scene transitions when consuming Scene.actions?

The `resolvePlaybackCursor` function in [`lib/choreography/index.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/choreography/index.ts) computes the next action across scene boundaries. When the current scene's `actions` array exhausts, the cursor automatically advances to the next scene's first action while preserving playback continuity and invoking scene change callbacks.

### What is the difference between fire-and-forget and synchronous actions in OpenMAIC?

Fire-and-forget actions (`spotlight`, `laser`) invoke `this.actionEngine.execute(action)` without an `await` statement, allowing immediate cursor advancement in the switch block (lines 862‑864). Synchronous actions (speech, video, whiteboard) use `await this.actionEngine.execute(action)` (lines 835‑842) to block progression until the action completes, ensuring temporal alignment between audio, visual, and interactive elements.

### Can I execute Scene.actions outside of the Playback Engine loop?

Yes. The ActionEngine is designed for independent use. Instantiate it with a StageStore and AudioPlayer, then call `await actionEngine.execute(action)` on any action object from `Scene.actions`. This is useful for preview modes, testing individual actions, or implementing custom playback controllers.

### Where are the action type definitions located in the OpenMAIC repository?

Type definitions for all 28+ supported actions reside in [`lib/types/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts). This includes interfaces like `SpeechAction`, `SpotlightAction`, `WhiteboardAction`, and `WidgetAction`. The `Scene` interface in [`lib/types/stage.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/stage.ts) declares the `actions: Action[]` property that the Playback Engine consumes.