# What Are the Supported Speech and Whiteboard Actions in OpenMAIC?

> Explore supported speech and whiteboard actions in OpenMAIC. Discover how JSON-serializable speech and vector-based whiteboard primitives enable synchronized multimedia playback and real-time collaboration.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: api-reference
- Published: 2026-09-05

---

**OpenMAIC uses JSON-serializable `speech` actions for text-to-speech narration and `whiteboard` actions for vector-based drawing primitives, both defined in the DSL core to enable synchronized multimedia playback and real-time collaboration.**

The THU-MAIC/OpenMAIC repository models interactive classroom content through a domain-specific language (DSL) that represents instructional sequences as discrete actions. Two foundational action types—**speech** and **whiteboard**—handle audio narration and visual drawing surfaces respectively. Understanding their schemas, required fields, and rendering pipelines is essential for extending the platform's content capabilities or building custom export tools.

## Speech Actions

Speech actions are non-visual timeline elements that trigger audio playback or text-to-speech (TTS) synthesis. These actions are always positioned relative to visual content to ensure narration synchronization.

### Core Schema and Required Fields

According to the `Action` interface in [`packages/@openmaic/dsl/src/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/packages/@openmaic/dsl/src/action.ts), every speech action requires:

- **`id`**: A unique string identifier for the action instance
- **`type`**: The literal string `'speech'`
- **`text`**: The narration content to be spoken or subtitled

The concrete shape of speech-specific fields is exported from [[`types/web-speech.d.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/types/web-speech.d.ts)](https://github.com/THU-MAIC/OpenMAIC/blob/main/types/web-speech.d.ts), which defines the following optional parameters:

- **`audioId`**: Reference to a pre-generated TTS asset stored in the system
- **`audioUrl`**: Fallback URL for streamed audio content
- **`language`**: Language code (e.g., `'en-US'`, `'zh-CN'`) passed to the synthesis engine

### Audio Pipeline and Timeline Integration

When the runtime encounters a speech action, it resolves audio through a prioritized lookup chain. First, it attempts to locate the asset by `audioId`; if absent, it falls back to `audioUrl`. If neither reference exists, the runtime submits the `text` field to the TTS engine to generate an audio blob dynamically. The system then schedules this audio in the video timeline and generates synchronized subtitle cues derived from the source text.

## Whiteboard Actions

Whiteboard actions capture complete snapshots of the drawing canvas as collections of vector primitives. These visual actions support both frame-by-frame playback rendering and real-time collaborative editing through diff-merge operations.

### Container Structure and Viewport State

Defined in the same [`packages/@openmaic/dsl/src/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/packages/@openmaic/dsl/src/action.ts) interface, whiteboard actions function as containers with the following structure:

- **`id`**: Unique identifier for the canvas state
- **`type`**: The literal string `'whiteboard'`
- **`items`**: An ordered array of `WhiteboardItem` drawing primitives
- **`backgroundColor`**: Optional hex string defining the canvas background (e.g., `#ffffff`)
- **`viewport`**: Optional `ViewportSpec` object specifying initial pan coordinates and zoom level

The rendering logic that processes these containers resides in [[`lib/whiteboard/viewport.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/viewport.ts)](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/viewport.ts), which handles rasterization and coordinate transformations during playback.

### Drawing Primitives

Individual drawing elements are defined in [[`lib/whiteboard/item.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/item.ts)](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/item.ts) as discriminated union types. The supported primitives include:

- **`WhiteboardLine`**: Vector lines with `start` and `end` coordinates, `color`, and `width` properties
- **`WhiteboardRect`**: Rectangle primitives with position and dimension fields
- **`WhiteboardEllipse`**: Circular and oval shapes
- **`WhiteboardText`**: Text labels with `position`, `content`, `fontSize`, and `color` attributes
- **`WhiteboardPath`**: Freehand paths and arrow shapes for complex diagrams

The engine consumes these items to produce vector graphics for video export and to synchronize live editing sessions between multiple participants.

## Practical Implementation Examples

The following TypeScript examples demonstrate the construction of both action types, following the patterns found in [[`tests/video-export/helpers.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/video-export/helpers.ts)](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/video-export/helpers.ts):

```typescript
// Speech action with pre-generated TTS asset
const narrationAction: Action = {
  id: "speech-intro-001",
  type: "speech",
  text: "Welcome to the lesson on photosynthesis.",
  audioId: "tts_welcome_001",
  language: "en-US"
};

// Whiteboard action with line and text primitives
const diagramAction: Action = {
  id: "wb-diagram-001",
  type: "whiteboard",
  backgroundColor: "#ffffff",
  items: [
    {
      type: "line",
      id: "sun-ray-1",
      start: { x: 100, y: 150 },
      end: { x: 300, y: 150 },
      color: "#0066ff",
      width: 4
    },
    {
      type: "text",
      id: "label-sunlight",
      position: { x: 200, y: 130 },
      content: "Sunlight",
      fontSize: 16,
      color: "#000000"
    }
  ]
};

// Composite scene combining both action types
const lessonScene = {
  id: "scene-001",
  type: "slide",
  actions: [narrationAction, diagramAction]
};

```

## Summary

- **Speech actions** in OpenMAIC handle audio narration through the `speech` type, supporting both pre-generated assets (`audioId`, `audioUrl`) and dynamic TTS generation from the required `text` field.
- **Whiteboard actions** use the `whiteboard` type to encapsulate vector drawing states, storing primitives like lines, rectangles, ellipses, text, and paths in the `items` array.
- Both action types are serializable JSON objects defined in `packages/@openmaic/dsl/src/action.ts`, making them suitable for network transmission and persistent storage in course manifests.
- The rendering pipeline references [`lib/whiteboard/viewport.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/viewport.ts) for canvas rasterization and [`lib/whiteboard/item.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/item.ts) for primitive definitions.

## Frequently Asked Questions

### How does OpenMAIC handle missing audio assets in speech actions?

When a speech action lacks both `audioId` and `audioUrl`, the runtime automatically submits the `text` content to the integrated TTS engine. The system generates a new audio blob, schedules it in the video timeline, and creates synchronized subtitle cues without requiring manual asset preparation.

### Can whiteboard actions be modified during collaborative sessions?

Yes. Whiteboard actions support diff-merge operations during collaborative editing, allowing participants to modify the `items` array in real-time. The engine synchronizes these changes between clients while maintaining vector quality for final video export.

### What file contains the TypeScript definitions for speech action fields?

Speech-specific type declarations—including optional fields like `audioId`, `audioUrl`, and `language`—are located in [`types/web-speech.d.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/types/web-speech.d.ts), while the core `Action` interface resides in `packages/@openmaic/dsl/src/action.ts`.

### How are whiteboard drawing primitives structured?

Each primitive is a `WhiteboardItem` union type defined in [`lib/whiteboard/item.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/item.ts). Primitives include `line`, `rect`, `ellipse`, `text`, and `path` types, each with geometry-specific fields (such as `start`/`end` coordinates for lines or `position` for text) that the rendering engine processes from [`lib/whiteboard/viewport.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/whiteboard/viewport.ts).