How the OpenMAIC Action Engine Handles Synchronous Speech Actions

The OpenMAIC ActionEngine treats speech as a synchronous action by returning a Promise that resolves only after audio playback completes, ensuring sequential execution of subsequent actions.

The ActionEngine serves as the central execution coordinator for all agent behaviors in the THU-MAIC/OpenMAIC framework. Unlike fire-and-forget actions such as spotlights or lasers, speech actions require precise synchronization to guarantee that narration finishes before subsequent visual cues appear. This article examines the internal mechanics of how the engine processes these synchronous speech actions according to the source code implementation.

Action Classification in the ActionEngine

The ActionEngine categorizes all incoming actions into two distinct execution models:

• Fire-and-forget – Actions like spotlight or laser dispatch immediately and return control to the caller without waiting for completion.
• Synchronous – Actions like speech, play_video, and whiteboard drawing operations return a Promise that settles only after the action fully completes.

This architectural distinction ensures that time-sensitive sequences—such as highlighting an element only after the narrator finishes speaking—execute in the correct order without race conditions.

The Speech Action Execution Flow

When the engine receives a speech action, it follows a rigorous four-step process to guarantee synchronous completion tracking.

Routing and Type Detection

In lib/action/engine.ts, the execute() method (lines 32‑47) inspects the incoming action type and routes speech payloads to the dedicated handler:

// Simplified routing logic from engine.ts
switch (action.type) {
  case 'speech':
    return this.executeSpeech(action);
  // ... other cases
}

The method returns the result of executeSpeech() directly, propagating the promise up to the caller.

Audio Player Validation

The executeSpeech() method first verifies that an AudioPlayer instance is available before attempting playback. If the audio infrastructure is missing, the method returns immediately to prevent runtime errors:

if (!this.audioPlayer) return;

Promise-Based Completion Handling

The core synchronization mechanism relies on JavaScript promises coupled with event listeners. The engine constructs a new Promise that resolves through an onEnded callback registered on the AudioPlayer instance:

// From lib/action/engine.ts around line 40
audioPlayer.onEnded(() => {
  resolve(); // Resolves only when audio finishes naturally
});

The engine then initiates playback using the speech action’s audioId (or legacy audioUrl). If playback fails to start, the promise resolves immediately (lines 44‑47) to prevent indefinite blocking:

const success = audioPlayer.play(action.audioId);
if (!success) {
  resolve(); // Immediate resolution on failure
}

Return Value and Caller Awaiting

The method returns a Promise<void> that callers can await. This design ensures that any code following a speech action—such as triggering a spotlight—waits for the audio duration to elapse before executing.

Practical Implementation Examples

The following patterns demonstrate how to leverage the synchronous nature of speech actions in your OpenMAIC applications.

Basic Speech Execution

This example initializes the engine with an AudioPlayer and awaits speech completion:

import { ActionEngine } from '@/lib/action/engine';
import { AudioPlayer } from '@/lib/utils/audio-player';

const audioPlayer = new AudioPlayer();
const engine = new ActionEngine(stageStore, audioPlayer);

const speechAction = {
  id: 'speech-1',
  type: 'speech',
  text: 'Welcome to the lesson!',
  audioId: 'audio-123',  // Pre-generated audio asset identifier
};

// Blocks until audio finishes playing
await engine.execute(speechAction);
console.log('Speech completed');

Chaining Dependent Actions

Because speech actions return promises, you can sequence visual cues to appear only after narration completes:

// Spotlight appears only after narration finishes
await engine.execute(speechAction);
engine.execute({ 
  id: 'spot-1', 
  type: 'spotlight', 
  elementId: 'elem-42' 
});

This pattern eliminates timing bugs where visual elements might appear before the corresponding audio explanation begins.

Key Source Files and Components

Understanding these three files provides complete visibility into the speech synchronization architecture:

• lib/action/engine.ts – Contains the ActionEngine class with the execute() routing logic and executeSpeech() method that implements the promise-based completion tracking.
• lib/types/action.ts – Defines the SpeechAction interface and type definitions that enforce the contract for speech payloads including audioId and text properties.
• lib/utils/audio-player.ts – Provides the audio abstraction layer that emits ended events, enabling the engine to detect when playback concludes.

Summary

• The ActionEngine classifies actions as either fire-and-forget or synchronous, with speech falling into the latter category.
• Synchronous speech actions return a Promise<void> that resolves via an onEnded callback registered on the AudioPlayer instance.
• The routing logic in execute() (lines 32‑47) forwards speech actions to executeSpeech(), which validates the audio player availability before initiating playback.
• Error handling ensures promises resolve immediately if playback fails to start, preventing application hangs.
• This architecture guarantees sequential action execution, allowing subsequent actions to wait for speech completion naturally using standard await semantics.

Frequently Asked Questions

What makes speech actions synchronous in OpenMAIC?

Speech actions are synchronous because the ActionEngine returns an active Promise rather than completing immediately. This promise remains pending until the AudioPlayer emits an ended event or encounters an error, allowing the JavaScript event loop to pause execution until the audio finishes.

How does the Action Engine sequence actions after speech?

Because engine.execute() returns a promise for speech actions, developers can use standard await syntax or .then() chains. The engine will not process subsequent actions in the queue until the previous speech promise resolves, creating an implicit barrier that preserves narrative flow.

What happens if the AudioPlayer fails to start playback?

If audioPlayer.play() returns a falsy value indicating startup failure, the executeSpeech() method immediately calls resolve() (lines 44‑47). This prevents the promise from remaining pending indefinitely while still allowing the action sequence to continue, treating the failure as a completed (though unsuccessful) operation.

Where are the speech action types defined?

The TypeScript interfaces for speech actions, including the SpeechAction type with its audioId and text properties, are defined in lib/types/action.ts. These type definitions ensure compile-time safety when constructing action payloads for the engine.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}