What Is the Role of the Playback Engine in OpenMAIC? A Deep Dive Into the Core Orchestrator

The Playback Engine in OpenMAIC is a state-machine-driven orchestrator that unifies automated lecture playback and live discussion modes, managing audio/TTS, user interruptions, and state persistence across idle, playing, paused, and live states.

The Playback Engine sits at the heart of the OpenMAIC educational platform, bridging static lecture content with dynamic, user-driven interactions. Whether delivering pre-recorded lessons via text-to-speech or enabling real-time discussions through ProactiveCards, this component ensures seamless transitions without losing synchronization. Below is a comprehensive breakdown of its architecture, responsibilities, and API based on the source code in THU-MAIC/OpenMAIC.

Playback Engine Modes and State Machine

The Playback Engine implements a finite state machine with four distinct modes defined in lib/playback/engine.ts lines 9-22:

• idle — No active playback; the engine awaits initialization
• playing — Automated lecture delivery with audio/TTS execution
• paused — Temporary halt preserving all timers and synthesis state
• live — Interactive discussion mode triggered by user engagement

Mode transitions are strictly controlled. For example, receiving a user message during playing triggers a save of the lecture cursor, audio termination, and a switch to live engine.ts#L138-L166.

Core Responsibilities of the Playback Engine

Direct Action Execution Without Compilation

Unlike systems that compile scenes into an intermediate format, the Playback Engine consumes Scene.actions[] directly through its ActionEngine dependency engine.ts#L4-L5. This design eliminates preprocessing overhead and allows dynamic modifications to lecture content.

Cursor Management and Snapshot Persistence

The engine maintains precise playback position through scene and action indices. Two critical methods enable state durability:

• getSnapshot() — Serializes sceneIndex, actionIndex, consumed discussion IDs, and sceneId
• restoreFromSnapshot() — Reconstructs engine state from a saved snapshot

These are implemented in engine.ts#L32-L40, supporting features like progress recovery across browser sessions.

Audio and TTS Coordination

The Playback Engine handles three audio delivery strategies engine.ts#L86-L106:

1. Pre-generated audio files — Played via AudioPlayer
2. Browser-native TTS — Using the Web Speech API with voice caching
3. Timed reading — Chunked synthesis for long text segments to prevent browser limitations

Voice list caching and chunked synthesis utilities appear in engine.ts#L150-L190, ensuring cross-browser compatibility for Chrome and Firefox.

Discussion Trigger Handling

Proactive discussions are surfaced through a delayed callback mechanism. When a discussion action becomes active, the engine waits 3 seconds then invokes onProactiveShow engine.ts#L98-L114. The markDiscussionConsumed method engine.ts#L48-L52 prevents re-triggering already-handled discussions.

User Interrupt Management

The handleUserInterrupt() method implements the full interruption protocol:

engine.handleUserInterrupt('What is the definition of AI?');

This executes engine.ts#L138-L166:

1. Saves current lecture cursor to lectureCursorBeforeInterrupt
2. Stops all audio/TTS immediately
3. Switches mode to live
4. Triggers onUserInterrupt callback for chat UI integration

When discussion ends, handleEndDiscussion resets the cursor and returns to idle engine.ts#L99-L114.

Public API and Lifecycle Callbacks

The Playback Engine exposes a clean control surface engine.ts#L49-L73 that shields UI layers from internal complexity:

Method	Purpose
start()	Begin playback from first scene/action
continuePlayback()	Resume after interruption using saved cursor
pause() / resume()	Temporary halt with full state preservation
stop()	Terminate playback and reset to idle
jumpToAction(index, options)	Navigate to specific action with optional autoplay

All state changes emit through constructor callbacks defined in types.ts lines 31-65:

• onModeChange — Engine mode transitions
• onSceneChange / onActionChange — Content progression
• onSpeechStart / onSpeechEnd — TTS lifecycle
• onEffectFire — Action effect execution
• onProgress — General progress updates

Safe Navigation and Jump Validation

The engine prevents illegal navigation through canJumpToAction engine.ts#L74-L80, which verifies that target actions lie within a reconstructable prefix. This ensures that jumping to arbitrary positions won't break lecture continuity or leave the engine in an inconsistent state.

The underlying logic resides in lib/playback/action-navigation.ts, with canJumpWithinReconstructablePrefix providing the mathematical guarantee.

Practical Code Examples

Engine Instantiation

import { PlaybackEngine } from '@/lib/playback/engine';
import { ActionEngine } from '@/lib/action/engine';
import { AudioPlayer } from '@/lib/utils/audio-player';

const engine = new PlaybackEngine(
  scenes,           // Scene[] array from lecture content
  new ActionEngine(),
  new AudioPlayer(),
  {
    onModeChange: (mode) => console.log('Mode →', mode),
    onProgress: (snap) => saveProgress(snap),
    onProactiveShow: (trigger) => showDiscussionCard(trigger),
    onSpeechStart: (text) => console.log('Speaking:', text),
    onUserInterrupt: (msg) => openChatInterface(msg),
  },
);

Basic Playback Control

// Start fresh lecture
engine.start();

// Pause and resume
engine.pause();
engine.resume();

// Continue after discussion ends
engine.continuePlayback();

State Persistence Pattern

// Save before page unload
window.addEventListener('beforeunload', () => {
  localStorage.setItem('playbackSnap', JSON.stringify(engine.getSnapshot()));
});

// Restore on application mount
const saved = localStorage.getItem('playbackSnap');
if (saved) {
  engine.restoreFromSnapshot(JSON.parse(saved));
}

Precision Navigation

// Jump to 5th action in first scene and auto-play
engine.jumpToAction(5, { autoplay: true });

// Check validity before attempting jump
if (engine.canJumpToAction(12)) {
  engine.jumpToAction(12);
}

Key Source Files in the Playback Module

File	Purpose	Critical Exports
lib/playback/engine.ts	Core state machine and public API	PlaybackEngine class
lib/playback/types.ts	Type definitions	PlaybackSnapshot, EngineMode, PlaybackEngineCallbacks
lib/playback/cursor.ts	Action resolution utilities	resolvePlaybackCursor
lib/playback/action-navigation.ts	Safe jump validation	canJumpWithinReconstructablePrefix
lib/playback/auto-resume.ts	Post-discussion resumption logic	Auto-resume determination
lib/playback/index.ts	Module exports	Barrel re-exports

Summary

• The Playback Engine orchestrates all lecture delivery and discussion flow in OpenMAIC through a rigorously defined four-mode state machine (idle, playing, paused, live)
• It executes actions directly without intermediate compilation, reducing latency and enabling dynamic content
• Cursor persistence via getSnapshot() and restoreFromSnapshot() enables seamless progress recovery across sessions
• User interruption handling preserves exact lecture position, switches to discussion mode, and supports automatic resumption
• The public API (start, pause, resume, jumpToAction, etc.) provides UI-agnostic control while lifecycle callbacks enable reactive application architecture

Frequently Asked Questions

What triggers the Playback Engine to switch to live mode?

A user-generated message or interaction triggers handleUserInterrupt(), which saves the current lecture cursor, stops all audio/TTS, and transitions the engine to live mode engine.ts#L138-L166. This ensures no content is lost when the user initiates discussion.

How does the Playback Engine handle long text that exceeds browser TTS limits?

The engine implements chunked speech synthesis engine.ts#L150-L190, breaking long text into browser-manageable segments while maintaining the illusion of continuous speech. It also caches the available voice list to avoid repeated permission prompts.

Can users jump to arbitrary positions in a lecture?

Only within validated boundaries. The jumpToAction() method checks canJumpToAction() engine.ts#L74-L80, which ensures targets lie within a reconstructable prefix—actions whose dependencies have all been satisfied. This prevents navigation to states that would break lecture logic.

What happens to playback progress if the browser crashes?

The getSnapshot() method produces a fully serializable state including scene index, action index, consumed discussion IDs, and scene identifier engine.ts#L32-L40. Applications can persist this to localStorage or a backend, then restore via restoreFromSnapshot() on reinitialization.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}