# What Actions Can the OpenMAIC Action Engine Execute?

> Explore 18 action types supported by the OpenMAIC action engine including text-to-speech video control whiteboard drawing and more Discover its versatile capabilities.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: api-reference
- Published: 2026-09-08

---

**The OpenMAIC action engine can execute 18 distinct action types**, including text-to-speech synthesis, video playback control, whiteboard drawing and editing, slide navigation, tool invocations, material upload and extraction, session lifecycle management, and error handling.

This guide provides the complete technical reference for the **Open Multi-Agent Interactive Classroom (OpenMAIC)** action engine. All actions are defined in the **OpenMAIC DSL** (`@openmaic/dsl`) as a discriminated-union type `Action`, with the engine dispatching each to its appropriate runtime handler based on the `type` field.

---

## Speech and Narration Actions

The OpenMAIC action engine generates spoken narration through **text-to-speech synthesis**.

| Action | Description | Key Parameters |
|--------|-------------|--------------|
| **`speech`** | Text-to-speech synthesis for AI narration or assistant voice | `id`, `text`, `voice?`, `audioUrl?` |

In `packages/@openmaic/dsl/src/action.ts`, the `speech` action type enables dynamic voice generation. The engine routes this to a TTS provider through the `generate_scene_actions` dispatcher in [`lib/server/agent-runtime/generation-tools.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/generation-tools.ts).

```typescript
import { Action } from '@openmaic/dsl';

const speak: Action = {
  type: 'speech',
  id: 'a-001',
  text: 'Welcome to the quantum physics lesson!',
  voice: 'en-US-Standard-B',
};

await generateSceneActions([speak]);

```

The optional `audioUrl` parameter allows pre-generated audio to be substituted for on-the-fly synthesis, improving latency for repeated content.

---

## Video Playback Actions

The action engine controls embedded video elements through three discrete action types.

| Action | Description |
|--------|-------------|
| **`play_video`** | Starts playback, optionally seeking to a timestamp |
| **`pause_video`** | Pauses a playing video |
| **`stop_video`** | Stops playback and resets to the start frame |

```typescript
const play: Action = {
  type: 'play_video',
  elementId: 'video_42',
  seekMs: 0,
};

await generateSceneActions([play]);

```

All video actions target specific elements via `elementId`, defined in the slide model. The runtime maps these to HTML5 video player controls.

---

## Whiteboard Drawing and Annotation Actions

The whiteboard subsystem supports six action types for interactive annotation, as implemented in the canvas renderer.

### Basic Drawing Actions

| Action | Description |
|--------|-------------|
| **`wb_draw_line`** | Draws straight lines between two points |
| **`wb_draw_text`** | Renders text blocks with font and color control |
| **`wb_draw_shape`** | Draws rectangles, ellipses, and polygons |

```typescript
const line: Action = {
  type: 'wb_draw_line',
  id: 'w-01',
  from: { x: 100, y: 150 },
  to:   { x: 400, y: 150 },
  color: '#ff5722',
  strokeWidth: 3,
};

await generateSceneActions([line]);

```

### Interactive Manipulation Actions

| Action | Description |
|--------|-------------|
| **`wb_erase`** | Removes content by region or object ID |
| **`wb_highlight`** | Temporary visual emphasis with duration |
| **`wb_focus`** | Camera movement to center on target with optional zoom |

The **`wb_highlight`** action creates transient attention cues, while **`wb_focus`** enables programmatic camera control for guided instruction.

---

## Navigation Actions

The action engine manages spatial and structural navigation through two action types.

| Action | Description |
|--------|-------------|
| **`navigate_slide`** | Switches to specific slide or directional movement |
| **`navigate_course`** | Jumps between courses or modules |

```typescript
const nav: Action = {
  type: 'navigate_slide',
  slideId: 'slide-07',
  direction: 'next'  // or 'prev'
};

```

These actions update the presentation state observable by all connected clients, maintaining synchronization across the collaborative classroom.

---

## Tool Invocation and Generation Actions

The **`generate_actions`** type enables the engine to call external skills and tools, making the system extensible.

| Action | Description |
|--------|-------------|
| **`generate_actions`** | Invokes a tool that returns new actions (e.g., slide generation, quiz creation) |
| **`tool_progress`** | Emits progress updates for long-running operations |
| **`tool_error`** | Signals tool failure with diagnostic information |

```typescript
const gen: Action = {
  type: 'generate_actions',
  toolName: 'slide_generator',
  toolDetails: { order: 1 },
  input: { topic: 'Newtonian Mechanics' },
};

await generateSceneActions([gen]);  // tool returns further actions for execution

```

The tool runtime in [`lib/server/agent-runtime/generation-tools.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/generation-tools.ts) handles the request-response cycle, streaming progress via `tool_progress` actions and capturing errors through `tool_error`.

---

## Action Modification Actions

The **`edit_actions`** type implements JSON-Patch-style mutation of existing actions.

| Action | Description |
|--------|-------------|
| **`edit_actions`** | Modifies an existing action's properties |

```typescript
const edit: Action = {
  type: 'edit_actions',
  targetActionId: 'a-001',
  patch: [{ op: 'replace', path: '/text', value: 'Let’s dive deeper.' }],
};

await generateSceneActions([edit]);

```

This pattern supports collaborative editing and AI-driven refinement, where generated content is subsequently revised without full regeneration. Test coverage appears in [`tests/workbench/tool-presentation.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/tool-presentation.test.ts).

---

## Material Management Actions

The engine handles learning content ingestion through three material-focused action types, as implemented in [`lib/workbench/material-upload-scheduling.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/material-upload-scheduling.ts) and related modules.

| Action | Description |
|--------|-------------|
| **`material_upload`** | Uploads PDF, audio, or video for later use |
| **`material_extraction`** | Initiates background processing (OCR, transcription) |
| **`edit_material`** | Updates metadata or replacement content |

The **`material_extraction`** action carries a `status` field with states: `running`, `finished`, or `failed`. Extraction flows are tested in [`tests/workbench/session-fold.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/session-fold.test.ts).

---

## Session Lifecycle and Resilience Actions

The action engine maintains durable workspace sessions with explicit lifecycle management, stored in [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts).

| Action | Description |
|--------|-------------|
| **`session_start`** | Initializes a new durable session |
| **`session_end`** | Gracefully terminates a session |
| **`session_cancel`** | Aborts an ongoing session |
| **`reconnect`** | Re-establishes WebSocket after transient disconnect |

```typescript
const reconnect: Action = {
  type: 'reconnect',
  sessionId: 'sess-uuid-1234'
};

```

These actions enable **resumable and steerable generation**—users can cancel, pause, or resume AI-driven content creation. The session API is defined in [`lib/workbench/workspace-actions.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/workspace-actions.ts) with test coverage in [`tests/workbench/owner-session-client.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/owner-session-client.test.ts).

---

## How the OpenMAIC Action Engine Processes Actions

Understanding the execution pipeline clarifies how these 18 action types operate:

1. **Parsing** — Incoming objects are validated against the `Action` union type in `packages/@openmaic/dsl/src/action.ts`

2. **Dispatch** — The `generate_scene_actions` function routes each action to its runtime: TTS service, HTML5 video player, whiteboard canvas, or skill backend

3. **Execution** — Side effects occur (API calls, state updates, rendering), producing result snapshots

4. **Persistence** — Session actions are written to the agent-session store, enabling recovery and replay

---

## Summary

- **18 action types** are available in the OpenMAIC action engine, defined in `packages/@openmaic/dsl/src/action.ts`
- **Speech and video** actions handle multimedia presentation
- **Whiteboard actions** (`wb_*`) provide six primitives for interactive annotation
- **Navigation actions** control slide and course structure
- **Tool actions** (`generate_actions`, `tool_progress`, `tool_error`) enable extensible AI skill integration
- **Modification actions** allow runtime patching of existing actions
- **Material actions** manage content ingestion and background processing
- **Session actions** provide durability, cancellation, and reconnect resilience

---

## Frequently Asked Questions

### What is the OpenMAIC action engine?

The OpenMAIC action engine is a runtime dispatcher that executes 18 distinct action types defined in the `@openmaic/dsl` package. It parses incoming action objects, routes them to appropriate handlers (TTS, video, whiteboard, tools), executes side effects, and persists session state for resumability.

### How are new action types added to OpenMAIC?

New action types require three changes: extend the `Action` union type in `packages/@openmaic/dsl/src/action.ts`, add a handler branch in [`lib/server/agent-runtime/generation-tools.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/generation-tools.ts), and implement the corresponding runtime module. The discriminated-union pattern with `type` field dispatch makes extension straightforward.

### Can OpenMAIC actions be modified after creation?

Yes, the **`edit_actions`** type supports JSON-Patch-style modifications to existing actions. Specify the `targetActionId` and an RFC 6902 compliant `patch` array to replace, add, or remove properties without regenerating the entire action sequence.

### How does OpenMAIC handle long-running tool execution?

The engine uses **`tool_progress`** actions to stream percentage and stage updates to clients, and **`tool_error`** to capture failures with structured diagnostic codes. Sessions can be cancelled mid-execution via **`session_cancel`**, with state preserved in [`lib/server/agent-runtime/store.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/server/agent-runtime/store.ts).