What Are the Supported Speech and Whiteboard Actions in OpenMAIC?

OpenMAIC uses JSON-serializable speech actions for text-to-speech narration and whiteboard actions for vector-based drawing primitives, both defined in the DSL core to enable synchronized multimedia playback and real-time collaboration.

The THU-MAIC/OpenMAIC repository models interactive classroom content through a domain-specific language (DSL) that represents instructional sequences as discrete actions. Two foundational action types—speech and whiteboard—handle audio narration and visual drawing surfaces respectively. Understanding their schemas, required fields, and rendering pipelines is essential for extending the platform's content capabilities or building custom export tools.

Speech Actions

Speech actions are non-visual timeline elements that trigger audio playback or text-to-speech (TTS) synthesis. These actions are always positioned relative to visual content to ensure narration synchronization.

Core Schema and Required Fields

According to the Action interface in packages/@openmaic/dsl/src/action.ts, every speech action requires:

• id: A unique string identifier for the action instance
• type: The literal string 'speech'
• text: The narration content to be spoken or subtitled

The concrete shape of speech-specific fields is exported from [types/web-speech.d.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/types/web-speech.d.ts), which defines the following optional parameters:

• audioId: Reference to a pre-generated TTS asset stored in the system
• audioUrl: Fallback URL for streamed audio content
• language: Language code (e.g., 'en-US', 'zh-CN') passed to the synthesis engine

Audio Pipeline and Timeline Integration

When the runtime encounters a speech action, it resolves audio through a prioritized lookup chain. First, it attempts to locate the asset by audioId; if absent, it falls back to audioUrl. If neither reference exists, the runtime submits the text field to the TTS engine to generate an audio blob dynamically. The system then schedules this audio in the video timeline and generates synchronized subtitle cues derived from the source text.

Whiteboard Actions

Whiteboard actions capture complete snapshots of the drawing canvas as collections of vector primitives. These visual actions support both frame-by-frame playback rendering and real-time collaborative editing through diff-merge operations.

Container Structure and Viewport State

Defined in the same packages/@openmaic/dsl/src/action.ts interface, whiteboard actions function as containers with the following structure:

• id: Unique identifier for the canvas state
• type: The literal string 'whiteboard'
• items: An ordered array of WhiteboardItem drawing primitives
• backgroundColor: Optional hex string defining the canvas background (e.g., #ffffff)
• viewport: Optional ViewportSpec object specifying initial pan coordinates and zoom level

The rendering logic that processes these containers resides in [lib/whiteboard/viewport.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/viewport.ts), which handles rasterization and coordinate transformations during playback.

Drawing Primitives

Individual drawing elements are defined in [lib/whiteboard/item.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/item.ts) as discriminated union types. The supported primitives include:

• WhiteboardLine: Vector lines with start and end coordinates, color, and width properties
• WhiteboardRect: Rectangle primitives with position and dimension fields
• WhiteboardEllipse: Circular and oval shapes
• WhiteboardText: Text labels with position, content, fontSize, and color attributes
• WhiteboardPath: Freehand paths and arrow shapes for complex diagrams

The engine consumes these items to produce vector graphics for video export and to synchronize live editing sessions between multiple participants.

Practical Implementation Examples

The following TypeScript examples demonstrate the construction of both action types, following the patterns found in [tests/video-export/helpers.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/video-export/helpers.ts):

// Speech action with pre-generated TTS asset
const narrationAction: Action = {
  id: "speech-intro-001",
  type: "speech",
  text: "Welcome to the lesson on photosynthesis.",
  audioId: "tts_welcome_001",
  language: "en-US"
};

// Whiteboard action with line and text primitives
const diagramAction: Action = {
  id: "wb-diagram-001",
  type: "whiteboard",
  backgroundColor: "#ffffff",
  items: [
    {
      type: "line",
      id: "sun-ray-1",
      start: { x: 100, y: 150 },
      end: { x: 300, y: 150 },
      color: "#0066ff",
      width: 4
    },
    {
      type: "text",
      id: "label-sunlight",
      position: { x: 200, y: 130 },
      content: "Sunlight",
      fontSize: 16,
      color: "#000000"
    }
  ]
};

// Composite scene combining both action types
const lessonScene = {
  id: "scene-001",
  type: "slide",
  actions: [narrationAction, diagramAction]
};

Summary

• Speech actions in OpenMAIC handle audio narration through the speech type, supporting both pre-generated assets (audioId, audioUrl) and dynamic TTS generation from the required text field.
• Whiteboard actions use the whiteboard type to encapsulate vector drawing states, storing primitives like lines, rectangles, ellipses, text, and paths in the items array.
• Both action types are serializable JSON objects defined in packages/@openmaic/dsl/src/action.ts, making them suitable for network transmission and persistent storage in course manifests.
• The rendering pipeline references lib/whiteboard/viewport.ts for canvas rasterization and lib/whiteboard/item.ts for primitive definitions.

Frequently Asked Questions

How does OpenMAIC handle missing audio assets in speech actions?

When a speech action lacks both audioId and audioUrl, the runtime automatically submits the text content to the integrated TTS engine. The system generates a new audio blob, schedules it in the video timeline, and creates synchronized subtitle cues without requiring manual asset preparation.

Can whiteboard actions be modified during collaborative sessions?

Yes. Whiteboard actions support diff-merge operations during collaborative editing, allowing participants to modify the items array in real-time. The engine synchronizes these changes between clients while maintaining vector quality for final video export.

What file contains the TypeScript definitions for speech action fields?

Speech-specific type declarations—including optional fields like audioId, audioUrl, and language—are located in types/web-speech.d.ts, while the core Action interface resides in packages/@openmaic/dsl/src/action.ts.

How are whiteboard drawing primitives structured?

Each primitive is a WhiteboardItem union type defined in lib/whiteboard/item.ts. Primitives include line, rect, ellipse, text, and path types, each with geometry-specific fields (such as start/end coordinates for lines or position for text) that the rendering engine processes from lib/whiteboard/viewport.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}