# What Is the Role of the Playback Engine in OpenMAIC? A Deep Dive Into the Core Orchestrator

> Discover the OpenMAIC playback engine role. This core orchestrator manages lecture playback, live discussions, audio, interruptions, and state persistence for seamless lecture delivery.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: deep-dive
- Published: 2026-09-08

---

**The Playback Engine in OpenMAIC is a state-machine-driven orchestrator that unifies automated lecture playback and live discussion modes, managing audio/TTS, user interruptions, and state persistence across `idle`, `playing`, `paused`, and `live` states.**

The Playback Engine sits at the heart of the OpenMAIC educational platform, bridging static lecture content with dynamic, user-driven interactions. Whether delivering pre-recorded lessons via text-to-speech or enabling real-time discussions through ProactiveCards, this component ensures seamless transitions without losing synchronization. Below is a comprehensive breakdown of its architecture, responsibilities, and API based on the source code in `THU-MAIC/OpenMAIC`.

## Playback Engine Modes and State Machine

The Playback Engine implements a finite state machine with four distinct modes defined in [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts) [lines 9-22](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L9-L22):

- **`idle`** — No active playback; the engine awaits initialization
- **`playing`** — Automated lecture delivery with audio/TTS execution
- **`paused`** — Temporary halt preserving all timers and synthesis state
- **`live`** — Interactive discussion mode triggered by user engagement

Mode transitions are strictly controlled. For example, receiving a user message during `playing` triggers a save of the lecture cursor, audio termination, and a switch to `live` [engine.ts#L138-L166](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L138-L166).

## Core Responsibilities of the Playback Engine

### Direct Action Execution Without Compilation

Unlike systems that compile scenes into an intermediate format, the Playback Engine consumes `Scene.actions[]` directly through its `ActionEngine` dependency [engine.ts#L4-L5](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L4-L5). This design eliminates preprocessing overhead and allows dynamic modifications to lecture content.

### Cursor Management and Snapshot Persistence

The engine maintains precise playback position through scene and action indices. Two critical methods enable state durability:

- `getSnapshot()` — Serializes `sceneIndex`, `actionIndex`, consumed discussion IDs, and `sceneId`
- `restoreFromSnapshot()` — Reconstructs engine state from a saved snapshot

These are implemented in [engine.ts#L32-L40](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L32-L40), supporting features like progress recovery across browser sessions.

### Audio and TTS Coordination

The Playback Engine handles three audio delivery strategies [engine.ts#L86-L106](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L86-L106):

1. **Pre-generated audio files** — Played via `AudioPlayer`
2. **Browser-native TTS** — Using the Web Speech API with voice caching
3. **Timed reading** — Chunked synthesis for long text segments to prevent browser limitations

Voice list caching and chunked synthesis utilities appear in [engine.ts#L150-L190](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L150-L190), ensuring cross-browser compatibility for Chrome and Firefox.

### Discussion Trigger Handling

Proactive discussions are surfaced through a delayed callback mechanism. When a discussion action becomes active, the engine waits 3 seconds then invokes `onProactiveShow` [engine.ts#L98-L114](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L98-L114). The `markDiscussionConsumed` method [engine.ts#L48-L52](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L48-L52) prevents re-triggering already-handled discussions.

### User Interrupt Management

The `handleUserInterrupt()` method implements the full interruption protocol:

```ts
engine.handleUserInterrupt('What is the definition of AI?');

```

This executes [engine.ts#L138-L166](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L138-L166):

1. Saves current lecture cursor to `lectureCursorBeforeInterrupt`
2. Stops all audio/TTS immediately
3. Switches mode to `live`
4. Triggers `onUserInterrupt` callback for chat UI integration

When discussion ends, `handleEndDiscussion` resets the cursor and returns to `idle` [engine.ts#L99-L114](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L99-L114).

## Public API and Lifecycle Callbacks

The Playback Engine exposes a clean control surface [engine.ts#L49-L73](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L49-L73) that shields UI layers from internal complexity:

| Method | Purpose |
|--------|---------|
| `start()` | Begin playback from first scene/action |
| `continuePlayback()` | Resume after interruption using saved cursor |
| `pause()` / `resume()` | Temporary halt with full state preservation |
| `stop()` | Terminate playback and reset to `idle` |
| `jumpToAction(index, options)` | Navigate to specific action with optional autoplay |

All state changes emit through constructor callbacks defined in [`types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/types.ts) [lines 31-65](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/types.ts#L31-L65):

- `onModeChange` — Engine mode transitions
- `onSceneChange` / `onActionChange` — Content progression
- `onSpeechStart` / `onSpeechEnd` — TTS lifecycle
- `onEffectFire` — Action effect execution
- `onProgress` — General progress updates

## Safe Navigation and Jump Validation

The engine prevents illegal navigation through `canJumpToAction` [engine.ts#L74-L80](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L74-L80), which verifies that target actions lie within a reconstructable prefix. This ensures that jumping to arbitrary positions won't break lecture continuity or leave the engine in an inconsistent state.

The underlying logic resides in [`lib/playback/action-navigation.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/action-navigation.ts), with `canJumpWithinReconstructablePrefix` providing the mathematical guarantee.

## Practical Code Examples

### Engine Instantiation

```ts
import { PlaybackEngine } from '@/lib/playback/engine';
import { ActionEngine } from '@/lib/action/engine';
import { AudioPlayer } from '@/lib/utils/audio-player';

const engine = new PlaybackEngine(
  scenes,           // Scene[] array from lecture content
  new ActionEngine(),
  new AudioPlayer(),
  {
    onModeChange: (mode) => console.log('Mode →', mode),
    onProgress: (snap) => saveProgress(snap),
    onProactiveShow: (trigger) => showDiscussionCard(trigger),
    onSpeechStart: (text) => console.log('Speaking:', text),
    onUserInterrupt: (msg) => openChatInterface(msg),
  },
);

```

### Basic Playback Control

```ts
// Start fresh lecture
engine.start();

// Pause and resume
engine.pause();
engine.resume();

// Continue after discussion ends
engine.continuePlayback();

```

### State Persistence Pattern

```ts
// Save before page unload
window.addEventListener('beforeunload', () => {
  localStorage.setItem('playbackSnap', JSON.stringify(engine.getSnapshot()));
});

// Restore on application mount
const saved = localStorage.getItem('playbackSnap');
if (saved) {
  engine.restoreFromSnapshot(JSON.parse(saved));
}

```

### Precision Navigation

```ts
// Jump to 5th action in first scene and auto-play
engine.jumpToAction(5, { autoplay: true });

// Check validity before attempting jump
if (engine.canJumpToAction(12)) {
  engine.jumpToAction(12);
}

```

## Key Source Files in the Playback Module

| File | Purpose | Critical Exports |
|------|---------|----------------|
| [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts) | Core state machine and public API | `PlaybackEngine` class |
| [`lib/playback/types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/types.ts) | Type definitions | `PlaybackSnapshot`, `EngineMode`, `PlaybackEngineCallbacks` |
| [`lib/playback/cursor.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/cursor.ts) | Action resolution utilities | `resolvePlaybackCursor` |
| [`lib/playback/action-navigation.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/action-navigation.ts) | Safe jump validation | `canJumpWithinReconstructablePrefix` |
| [`lib/playback/auto-resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/auto-resume.ts) | Post-discussion resumption logic | Auto-resume determination |
| [`lib/playback/index.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/index.ts) | Module exports | Barrel re-exports |

## Summary

- The **Playback Engine** orchestrates all lecture delivery and discussion flow in OpenMAIC through a rigorously defined **four-mode state machine** (`idle`, `playing`, `paused`, `live`)
- It **executes actions directly** without intermediate compilation, reducing latency and enabling dynamic content
- **Cursor persistence** via `getSnapshot()` and `restoreFromSnapshot()` enables seamless progress recovery across sessions
- **User interruption handling** preserves exact lecture position, switches to discussion mode, and supports automatic resumption
- The **public API** (`start`, `pause`, `resume`, `jumpToAction`, etc.) provides UI-agnostic control while **lifecycle callbacks** enable reactive application architecture

## Frequently Asked Questions

### What triggers the Playback Engine to switch to `live` mode?

A user-generated message or interaction triggers `handleUserInterrupt()`, which saves the current lecture cursor, stops all audio/TTS, and transitions the engine to `live` mode [engine.ts#L138-L166](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L138-L166). This ensures no content is lost when the user initiates discussion.

### How does the Playback Engine handle long text that exceeds browser TTS limits?

The engine implements **chunked speech synthesis** [engine.ts#L150-L190](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L150-L190), breaking long text into browser-manageable segments while maintaining the illusion of continuous speech. It also caches the available voice list to avoid repeated permission prompts.

### Can users jump to arbitrary positions in a lecture?

Only within validated boundaries. The `jumpToAction()` method checks `canJumpToAction()` [engine.ts#L74-L80](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L74-L80), which ensures targets lie within a **reconstructable prefix**—actions whose dependencies have all been satisfied. This prevents navigation to states that would break lecture logic.

### What happens to playback progress if the browser crashes?

The `getSnapshot()` method produces a fully serializable state including scene index, action index, consumed discussion IDs, and scene identifier [engine.ts#L32-L40](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts#L32-L40). Applications can persist this to `localStorage` or a backend, then restore via `restoreFromSnapshot()` on reinitialization.