How the OpenMAIC Playback Engine Works as a State Machine: A Deep Dive into the Core Architecture

The OpenMAIC playback engine is a deterministic finite-state machine with four core modes—idle, playing, paused, and live—that manages lecture playback, real-time discussions, and user interrupts through guarded async transitions.

The PlaybackEngine in THU-MAIC/OpenMAIC drives the entire interactive lecture experience. Understanding how this state machine operates is essential for anyone building custom integrations or debugging playback behavior. This article examines the implementation in lib/playback/engine.ts, breaking down each state, transition, and guard mechanism.

Core State Machine Modes

The engine operates in four mutually exclusive engine modes, defined in lib/playback/types.ts and managed centrally in lib/playback/engine.ts (lines 7–24):

Mode	Purpose	Typical Entry
idle	No active playback; system ready to start	start(), continuePlayback(), or after stop()
playing	Automatic lecture progression through scenes and actions	Transition from idle or resume()
paused	Playback frozen, timers cancelled, resumable	pause() from playing or live
live	Real-time user discussion after joining a ProactiveCard	confirmDiscussion() or handleUserInterrupt()

The state diagram embedded in the source header captures the primary flow:

idle ──► playing ──► paused
  ▲           ▲          │
  │           │ resume()│
  │           └─────────┘
  │
  │ handleEndDiscussion()
  │               │
  └───── live ────► paused

State Transitions and Their Implementation

Every state change occurs through explicit methods that enforce valid transitions. Invalid transitions are prevented by guard conditions, not by throwing errors—this makes the engine resilient to race conditions.

Starting Playback: idle → playing

Two entry points initialize playback from idle:

• start() (lines 49–61): Full reset. Clears all scene and action indices, sets mode to playing, then invokes processNext() to begin the action loop. Use this for fresh lectures.

• continuePlayback() (lines 63–71): Resume without reset. Preserves the existing cursor position, allowing a lecture to continue after a discussion ends. This is the internal mechanism handleEndDiscussion() uses.

Pausing and Resuming: playing ↔ paused

The pause() method (lines 21–60) performs several critical cleanup operations:

1. Cancels all pending timers via invalidatePlaybackGeneration().
2. Freezes audio playback and TTS synthesis.
3. Stores remaining speech time for accurate resume.
4. Sets mode to paused.

The resume() method (lines 63–84) reverses this:

1. Restores audio/TTS state or restarts reading timers.
2. Re-speaks any paused TTS chunks if using browser-native TTS.
3. Returns to playing—or transitions to live if currentTopicState === 'pending' (lines 70–74).

Entering Live Discussion: playing/live → live

Discussions interrupt normal flow through two paths:

• confirmDiscussion() (lines 53–73): User accepts a ProactiveCard invitation. The engine:

  • Marks the discussion trigger as consumed.
  • Saves the current lecture cursor via savedPlaybackCursor.
  • Sets mode to live.
  • Fires onDiscussionConfirmed callback.

• handleUserInterrupt() (lines 40–68): User sends a message during playback. More forceful:

  • Saves lecture position immediately.
  • Aborts any pending triggerDelayTimer.
  • Switches to live and stops all audio/TTS.

The skipDiscussion() path (lines 84–96) provides an alternative: mark the discussion consumed, clear the trigger, and continue playback without entering live.

Exiting Live Discussion: live → idle

When a discussion ends, handleEndDiscussion() (lines 99–115) performs cleanup:

1. Restores savedPlaybackCursor to resume the lecture.
2. Clears whiteboard state accumulated during discussion.
3. Fires onDiscussionEnd callback.
4. Returns engine to idle (not directly to playing; continuePlayback() is expected next).

Error paths use handleDiscussionError() (lines 21–37), which restores state and returns to idle without normal completion semantics.

Emergency Stop: any → idle

The stop() method (lines 16–34) force-terminates regardless of current mode:

• Resets all indices to zero.
• Clears all timers and audio state.
• Sets mode to idle.

Generation Guard: Preventing Stale Callbacks

Async JavaScript is prone to race conditions. The OpenMAIC playback engine solves this with a generation counter:

// From lib/playback/engine.ts, lines 93–98
private playbackGeneration = 0;

invalidatePlaybackGeneration(): void {
  this.playbackGeneration++;
}

isCurrentGeneration(gen: number): boolean {
  return gen === this.playbackGeneration;
}

Every async operation—audioPlayer.onEnded, TTS onend, setTimeout callbacks—receives the current generation number at creation time. Before executing any side effect, the callback validates isCurrentGeneration(generation). If pause(), stop(), or a new user action has incremented playbackGeneration, the stale callback exits silently.

This pattern appears throughout processNext() and the TTS chunking logic, ensuring that a pause-click at 0:30 doesn't trigger speech events from a timer set at 0:15.

The Action Processing Loop: processNext()

processNext() (lines 52–124) is the engine's heartbeat. It executes repeatedly via queueMicrotask until playback completes or is interrupted.

The method follows this sequence:

1. Guard check: Verify mode is still playing; exit otherwise.
2. Scene detection: If scene changed, fire onSceneChange.
3. Cursor resolution: Call resolvePlaybackCursor() (defined in lib/playback/cursor.ts, lines 41–46) to get the next action.
4. Progress snapshot: Emit onProgress before advancing the cursor—this ensures resume replays the same action, not the next one.
5. Action dispatch: Branch on action.type:

Action Type	Behavior
speech	Play pre-generated audio, or fall back to browser TTS/reading timer
spotlight / laser	Fire-and-forget visual effects, immediate queueMicrotask(processNext)
discussion	Set triggerDelayTimer for DISCUSSION_TRIGGER_DELAY_MS, pause until user response
whiteboard / widget	Synchronous execution via actionEngine

When the cursor exhausts all scenes, the engine sets mode to idle and fires onComplete (lines 67–71).

Browser-Native TTS Chunking

For environments without pre-generated audio, the engine supports browser-native TTS with a critical workaround: Chrome cuts off synthesis after ~15 seconds. The engine implements sentence-level chunking:

// From lib/playback/engine.ts, lines 57–66
splitIntoChunks(text: string): string[] {
  // Split on sentence boundaries, respecting quoted strings
  return text.match(/[^.!?]+[.!?]+(?:\s|$)|[^.!?]+$/g) || [text];
}

• playBrowserTTSChunk() (lines 84–95): Speaks one chunk, queues the next on onend.
• Pause behavior: Stores remaining chunks in browserTTSPausedChunks, cancels synthesis.
• Resume behavior: Re-speaks stored chunks via resume() (lines 80–87).

Callback Integration

The engine communicates with the UI through callbacks supplied at construction:

interface PlaybackCallbacks {
  onModeChange: (mode: EngineMode) => void;      // idle | playing | paused | live
  onSceneChange: (sceneId: string) => void;
  onSpeakerChange: (speakerId: string) => void;
  onProgress: (snapshot: PlaybackSnapshot) => void;
  onSpeechStart: (text: string) => void;
  onSpeechEnd: () => void;
  onProactiveShow: (trigger: DiscussionTrigger) => void;
  onProactiveHide: () => void;
  onDiscussionConfirmed: (question: string, prompt: string, agentId: string) => void;
  onDiscussionEnd: () => void;
  onUserInterrupt: () => void;
  getPlaybackSpeed: () => number;
}

These are invoked at precise state transition moments—onModeChange fires immediately after the mode variable updates, onProgress fires before cursor advancement for accurate resume positioning.

Practical Integration Example

import { PlaybackEngine } from '@/lib/playback/engine';
import { ActionEngine } from '@/lib/action/engine';
import { AudioPlayer } from '@/lib/utils/audio-player';

const scenes = await fetchLectureScenes();
const engine = new PlaybackEngine(scenes, new ActionEngine(), new AudioPlayer(), {
  onModeChange: (mode) => updateUIHeader(mode),
  onProactiveShow: (trigger) => showDiscussionCard(trigger),
  onDiscussionConfirmed: (q, p, id) => openLiveChat(q, p, id),
});

// Start lecture
engine.start();

// User controls
document.getElementById('pause')!.onclick = () => engine.pause();
document.getElementById('resume')!.onclick = () => engine.resume();

// Join discussion from UI
document.getElementById('join-chat')!.onclick = () => engine.confirmDiscussion();

// End discussion
document.getElementById('close-chat')!.onclick = () => {
  engine.handleEndDiscussion();
  engine.continuePlayback();  // Return to lecture
};

Supporting Files in the Playback System

File	Responsibility
lib/playback/engine.ts	Core state machine, transitions, processNext() loop
lib/playback/types.ts	EngineMode, PlaybackSnapshot, callback interfaces
lib/playback/cursor.ts	resolvePlaybackCursor—maps indices to actions
lib/playback/action-navigation.ts	Jump validation (canJumpWithinReconstructablePrefix)
lib/playback/auto-resume.ts	Inactivity detection and automatic resume
lib/action/engine.ts	Visual effect execution (spotlight, laser, whiteboard)

Summary

• The OpenMAIC playback engine implements a deterministic finite-state machine with four modes: idle, playing, paused, and live.
• All state transitions occur through explicit methods in lib/playback/engine.ts, with guard conditions preventing invalid paths.
• The generation counter pattern eliminates race conditions between async callbacks and user interrupts.
• processNext() drives continuous action execution, dispatching to audio, visual, or discussion handlers based on action type.
• Browser TTS chunking works around browser limitations while maintaining pause/resume accuracy.
• Callbacks provide precise UI synchronization at every state boundary.

Frequently Asked Questions

What happens if pause() is called while a discussion timer is pending?

The pause() method (lines 52–58) stores currentTopicState = 'pending' and sets mode to paused. When resume() executes, it detects this pending state and transitions to live instead of returning to playing (lines 70–74). The discussion ProactiveCard remains available for user interaction.

Can the engine jump to arbitrary positions in a lecture?

Yes, via jumpToAction(index, { autoplay }). However, not all positions are valid resume points. The lib/playback/action-navigation.ts module provides canJumpWithinReconstructablePrefix() to determine whether the target action can be correctly reconstructed from initial state—critical for actions that depend on accumulated whiteboard or widget state.

How does the engine handle audio playback across different browsers?

The AudioPlayer abstraction in lib/utils/audio-player.ts wraps pre-generated audio playback. For fallback, the engine detects the absence of pre-generated assets and routes to playBrowserTTSChunk(), which uses the Web Speech API with sentence-level chunking to avoid Chrome's 15-second cutoff limitation.

What triggers the automatic live transition during resume?

When resume() executes (lines 63–84), it checks if (this.currentTopicState === 'pending') before restoring playback. If a discussion was pending when paused, the engine switches to live mode and fires onProactiveShow again, allowing the user to join the discussion that was interrupted.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}