# How the OpenMAIC Playback Engine Works as a State Machine: A Deep Dive into the Core Architecture

> Explore the OpenMAIC playback engine's state machine architecture. Understand its four core modes idle playing paused and live managing lecture playback discussions and interrupts via guarded async transitions.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: deep-dive
- Published: 2026-09-05

---

**The OpenMAIC playback engine is a deterministic finite-state machine with four core modes—`idle`, `playing`, `paused`, and `live`—that manages lecture playback, real-time discussions, and user interrupts through guarded async transitions.**

The **PlaybackEngine** in [THU-MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC/) drives the entire interactive lecture experience. Understanding how this state machine operates is essential for anyone building custom integrations or debugging playback behavior. This article examines the implementation in [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts), breaking down each state, transition, and guard mechanism.

## Core State Machine Modes

The engine operates in four mutually exclusive **engine modes**, defined in [`lib/playback/types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/types.ts) and managed centrally in [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts) (lines 7–24):

| Mode | Purpose | Typical Entry |
|------|---------|-------------|
| `idle` | No active playback; system ready to start | `start()`, `continuePlayback()`, or after `stop()` |
| `playing` | Automatic lecture progression through scenes and actions | Transition from `idle` or `resume()` |
| `paused` | Playback frozen, timers cancelled, resumable | `pause()` from `playing` or `live` |
| `live` | Real-time user discussion after joining a ProactiveCard | `confirmDiscussion()` or `handleUserInterrupt()` |

The state diagram embedded in the source header captures the primary flow:

```

idle ──► playing ──► paused
  ▲           ▲          │
  │           │ resume()│
  │           └─────────┘
  │
  │ handleEndDiscussion()
  │               │
  └───── live ────► paused

```

## State Transitions and Their Implementation

Every state change occurs through explicit methods that enforce valid transitions. Invalid transitions are prevented by guard conditions, not by throwing errors—this makes the engine resilient to race conditions.

### Starting Playback: `idle → playing`

Two entry points initialize playback from `idle`:

- **`start()`** (lines 49–61): Full reset. Clears all scene and action indices, sets mode to `playing`, then invokes `processNext()` to begin the action loop. Use this for fresh lectures.

- **`continuePlayback()`** (lines 63–71): Resume without reset. Preserves the existing cursor position, allowing a lecture to continue after a discussion ends. This is the internal mechanism `handleEndDiscussion()` uses.

### Pausing and Resuming: `playing ↔ paused`

The **`pause()`** method (lines 21–60) performs several critical cleanup operations:

1. Cancels all pending timers via `invalidatePlaybackGeneration()`.
2. Freezes audio playback and TTS synthesis.
3. Stores remaining speech time for accurate resume.
4. Sets mode to `paused`.

The **`resume()`** method (lines 63–84) reverses this:

1. Restores audio/TTS state or restarts reading timers.
2. Re-speaks any paused TTS chunks if using browser-native TTS.
3. Returns to `playing`—or transitions to `live` if `currentTopicState === 'pending'` (lines 70–74).

### Entering Live Discussion: `playing/live → live`

Discussions interrupt normal flow through two paths:

- **`confirmDiscussion()`** (lines 53–73): User accepts a ProactiveCard invitation. The engine:
  - Marks the discussion trigger as consumed.
  - Saves the current lecture cursor via `savedPlaybackCursor`.
  - Sets mode to `live`.
  - Fires `onDiscussionConfirmed` callback.

- **`handleUserInterrupt()`** (lines 40–68): User sends a message during playback. More forceful:
  - Saves lecture position immediately.
  - Aborts any pending `triggerDelayTimer`.
  - Switches to `live` and stops all audio/TTS.

The **`skipDiscussion()`** path (lines 84–96) provides an alternative: mark the discussion consumed, clear the trigger, and continue playback without entering `live`.

### Exiting Live Discussion: `live → idle`

When a discussion ends, **`handleEndDiscussion()`** (lines 99–115) performs cleanup:

1. Restores `savedPlaybackCursor` to resume the lecture.
2. Clears whiteboard state accumulated during discussion.
3. Fires `onDiscussionEnd` callback.
4. Returns engine to `idle` (not directly to `playing`; `continuePlayback()` is expected next).

Error paths use **`handleDiscussionError()`** (lines 21–37), which restores state and returns to `idle` without normal completion semantics.

### Emergency Stop: `any → idle`

The **`stop()`** method (lines 16–34) force-terminates regardless of current mode:

- Resets all indices to zero.
- Clears all timers and audio state.
- Sets mode to `idle`.

## Generation Guard: Preventing Stale Callbacks

Async JavaScript is prone to race conditions. The OpenMAIC playback engine solves this with a **generation counter**:

```typescript
// From lib/playback/engine.ts, lines 93–98
private playbackGeneration = 0;

invalidatePlaybackGeneration(): void {
  this.playbackGeneration++;
}

isCurrentGeneration(gen: number): boolean {
  return gen === this.playbackGeneration;
}

```

Every async operation—`audioPlayer.onEnded`, TTS `onend`, `setTimeout` callbacks—receives the current generation number at creation time. Before executing any side effect, the callback validates `isCurrentGeneration(generation)`. If `pause()`, `stop()`, or a new user action has incremented `playbackGeneration`, the stale callback exits silently.

This pattern appears throughout `processNext()` and the TTS chunking logic, ensuring that a pause-click at 0:30 doesn't trigger speech events from a timer set at 0:15.

## The Action Processing Loop: `processNext()`

`processNext()` (lines 52–124) is the engine's heartbeat. It executes repeatedly via `queueMicrotask` until playback completes or is interrupted.

The method follows this sequence:

1. **Guard check**: Verify mode is still `playing`; exit otherwise.
2. **Scene detection**: If scene changed, fire `onSceneChange`.
3. **Cursor resolution**: Call `resolvePlaybackCursor()` (defined in [`lib/playback/cursor.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/cursor.ts), lines 41–46) to get the next action.
4. **Progress snapshot**: Emit `onProgress` **before** advancing the cursor—this ensures resume replays the same action, not the next one.
5. **Action dispatch**: Branch on `action.type`:

| Action Type | Behavior |
|-------------|----------|
| `speech` | Play pre-generated audio, or fall back to browser TTS/reading timer |
| `spotlight` / `laser` | Fire-and-forget visual effects, immediate `queueMicrotask(processNext)` |
| `discussion` | Set `triggerDelayTimer` for `DISCUSSION_TRIGGER_DELAY_MS`, pause until user response |
| `whiteboard` / `widget` | Synchronous execution via `actionEngine` |

When the cursor exhausts all scenes, the engine sets mode to `idle` and fires `onComplete` (lines 67–71).

## Browser-Native TTS Chunking

For environments without pre-generated audio, the engine supports browser-native TTS with a critical workaround: Chrome cuts off synthesis after ~15 seconds. The engine implements sentence-level chunking:

```typescript
// From lib/playback/engine.ts, lines 57–66
splitIntoChunks(text: string): string[] {
  // Split on sentence boundaries, respecting quoted strings
  return text.match(/[^.!?]+[.!?]+(?:\s|$)|[^.!?]+$/g) || [text];
}

```

- **`playBrowserTTSChunk()`** (lines 84–95): Speaks one chunk, queues the next on `onend`.
- **Pause behavior**: Stores remaining chunks in `browserTTSPausedChunks`, cancels synthesis.
- **Resume behavior**: Re-speaks stored chunks via `resume()` (lines 80–87).

## Callback Integration

The engine communicates with the UI through callbacks supplied at construction:

```typescript
interface PlaybackCallbacks {
  onModeChange: (mode: EngineMode) => void;      // idle | playing | paused | live
  onSceneChange: (sceneId: string) => void;
  onSpeakerChange: (speakerId: string) => void;
  onProgress: (snapshot: PlaybackSnapshot) => void;
  onSpeechStart: (text: string) => void;
  onSpeechEnd: () => void;
  onProactiveShow: (trigger: DiscussionTrigger) => void;
  onProactiveHide: () => void;
  onDiscussionConfirmed: (question: string, prompt: string, agentId: string) => void;
  onDiscussionEnd: () => void;
  onUserInterrupt: () => void;
  getPlaybackSpeed: () => number;
}

```

These are invoked at precise state transition moments—`onModeChange` fires immediately after the mode variable updates, `onProgress` fires before cursor advancement for accurate resume positioning.

## Practical Integration Example

```typescript
import { PlaybackEngine } from '@/lib/playback/engine';
import { ActionEngine } from '@/lib/action/engine';
import { AudioPlayer } from '@/lib/utils/audio-player';

const scenes = await fetchLectureScenes();
const engine = new PlaybackEngine(scenes, new ActionEngine(), new AudioPlayer(), {
  onModeChange: (mode) => updateUIHeader(mode),
  onProactiveShow: (trigger) => showDiscussionCard(trigger),
  onDiscussionConfirmed: (q, p, id) => openLiveChat(q, p, id),
});

// Start lecture
engine.start();

// User controls
document.getElementById('pause')!.onclick = () => engine.pause();
document.getElementById('resume')!.onclick = () => engine.resume();

// Join discussion from UI
document.getElementById('join-chat')!.onclick = () => engine.confirmDiscussion();

// End discussion
document.getElementById('close-chat')!.onclick = () => {
  engine.handleEndDiscussion();
  engine.continuePlayback();  // Return to lecture
};

```

## Supporting Files in the Playback System

| File | Responsibility |
|------|--------------|
| [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts) | Core state machine, transitions, `processNext()` loop |
| [`lib/playback/types.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/types.ts) | `EngineMode`, `PlaybackSnapshot`, callback interfaces |
| [`lib/playback/cursor.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/cursor.ts) | `resolvePlaybackCursor`—maps indices to actions |
| [`lib/playback/action-navigation.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/action-navigation.ts) | Jump validation (`canJumpWithinReconstructablePrefix`) |
| [`lib/playback/auto-resume.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/auto-resume.ts) | Inactivity detection and automatic resume |
| [`lib/action/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/action/engine.ts) | Visual effect execution (spotlight, laser, whiteboard) |

## Summary

- The OpenMAIC playback engine implements a **deterministic finite-state machine** with four modes: `idle`, `playing`, `paused`, and `live`.
- All state transitions occur through explicit methods in [`lib/playback/engine.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/engine.ts), with guard conditions preventing invalid paths.
- The **generation counter** pattern eliminates race conditions between async callbacks and user interrupts.
- **`processNext()`** drives continuous action execution, dispatching to audio, visual, or discussion handlers based on action type.
- **Browser TTS chunking** works around browser limitations while maintaining pause/resume accuracy.
- Callbacks provide precise UI synchronization at every state boundary.

## Frequently Asked Questions

### What happens if pause() is called while a discussion timer is pending?

The `pause()` method (lines 52–58) stores `currentTopicState = 'pending'` and sets mode to `paused`. When `resume()` executes, it detects this pending state and transitions to `live` instead of returning to `playing` (lines 70–74). The discussion ProactiveCard remains available for user interaction.

### Can the engine jump to arbitrary positions in a lecture?

Yes, via `jumpToAction(index, { autoplay })`. However, not all positions are valid resume points. The [`lib/playback/action-navigation.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/action-navigation.ts) module provides `canJumpWithinReconstructablePrefix()` to determine whether the target action can be correctly reconstructed from initial state—critical for actions that depend on accumulated whiteboard or widget state.

### How does the engine handle audio playback across different browsers?

The `AudioPlayer` abstraction in [`lib/utils/audio-player.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/utils/audio-player.ts) wraps pre-generated audio playback. For fallback, the engine detects the absence of pre-generated assets and routes to `playBrowserTTSChunk()`, which uses the Web Speech API with sentence-level chunking to avoid Chrome's 15-second cutoff limitation.

### What triggers the automatic live transition during resume?

When `resume()` executes (lines 63–84), it checks `if (this.currentTopicState === 'pending')` before restoring playback. If a discussion was pending when paused, the engine switches to `live` mode and fires `onProactiveShow` again, allowing the user to join the discussion that was interrupted.