# What Types of Actions Are Supported by the OpenMAIC Orchestration System?

> Discover the diverse action types supported by the OpenMAIC orchestration system, including speech synthesis, video playback, and session control. Learn how OpenMAIC streamlines complex workflows.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: api-reference
- Published: 2026-09-11

---

**The OpenMAIC orchestration system supports 28+ distinct action types—including speech synthesis, video playback, whiteboard drawing, and session control—defined centrally in `packages/@openmaic/dsl/src/action.ts` and executed through a LangGraph-based action engine.**

The THU-MAIC/OpenMAIC repository implements a sophisticated orchestration layer that manages AI-driven educational sessions through a deterministic **Action Engine**. This system recognizes diverse action types that enable agents to manipulate multimedia content, render interactive whiteboard primitives, and control application flow in real time.

## Action Types Supported by the OpenMAIC Orchestration System

The **Action Engine** categorizes its supported operations into nine functional groups. These definitions reside in [`packages/@openmaic/dsl/src/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/packages/@openmaic/dsl/src/action.ts), which serves as the central TypeScript discriminant union for all possible operations.

### Speech and Audio Actions

Audio narration capabilities include:

- **`speech`** – Initiates agent dialogue with natural language text
- **`tts`** – Direct text-to-speech synthesis requests
- **`play_audio`** – Streams pre-recorded audio assets to the client

### Video Control Actions

The system manages embedded video content through:

- **`play_video`** – Starts playback from a specified millisecond offset
- **`pause_video`** – Suspends active video streams
- **`seek_video`** – Repositions playback to a specific timestamp

### Whiteboard Drawing Actions

Vector graphics rendering supports seven distinct primitives:

- **`draw_line`** – Renders linear strokes between coordinates
- **`draw_rect`** – Creates rectangular shapes with fill and stroke properties
- **`draw_ellipse`** – Generates elliptical forms
- **`draw_path`** – Executes complex SVG path commands
- **`draw_text`** – Places typographic elements on the canvas
- **`draw_shape`** – Generic shape instantiation
- **`draw_chart`** – Data visualization rendering

### Spotlight and Laser Actions

Presentation aids include:

- **`spotlight`** – Focuses user attention on specific screen regions
- **`laser_pointer`** – Simulates laser pointer movement across content

### Material Handling Actions

Content management operations:

- **`material_extraction`** – Retrieves learning materials from storage
- **`material_upload`** – Persists user-generated content
- **`material_download`** – Delivers resources to local client storage

### Generation Helper Actions

Meta-operations for dynamic scene construction:

- **`generate_actions`** – Creates action sequences programmatically
- **`edit_actions`** – Modifies existing action chains
- **`generate_scene`** – Constructs complete scene definitions
- **`edit_scene`** – Updates active scene configurations

### Control and Flow Actions

Session state management primitives:

- **`wait`** – Blocks execution until conditions are met
- **`delay`** – Introduces timed pauses
- **`cancel`** – Terminates pending operations
- **`reconnect`** – Restores dropped client connections

### UI Manipulation Actions

Interface state transitions:

- **`expand_folder`** – Expands hierarchical navigation nodes
- **`collapse_folder`** – Collapses directory structures
- **`select_tab`** – Activates specific interface tabs
- **`open_course`** – Loads course-level content containers

### Miscellaneous Actions

Utility operations including **`highlight`**, **`show_hint`**, and **`raise_event`** for custom signaling.

## Action Execution Pipeline and Architecture

The OpenMAIC orchestration system processes these action types through a deterministic pipeline that ensures consistent execution and replayability.

### Definition Layer

The DSL package (`packages/@openmaic/dsl/src/action.ts`) defines the discriminated union of all **28+ action kinds** using TypeScript's type system. Each action carries a unique `id` and `type` property that routes it through the system.

### Orchestration Graph

The **LangGraph**-based orchestration layer (`lib/orchestration/`) routes agent turns and schedules action dispatch. This graph maintains session state and determines the execution order of operations, allowing the system to **resume, cancel, or steer** running sessions while preserving an event log.

### Runtime and Execution

The **agent runtime** (`lib/server/agent-runtime/`) consumes action definitions and manages durable session execution. Actions pass to the **action executor** (`lib/action/`), which contains concrete handlers for each type. For example, the speech handler streams TTS audio to the client, while whiteboard handlers render SVG primitives.

### Frontend Rendering

Client-side components in `components/whiteboard/` and `components/scene-renderers/` deserialize actions and update the UI. The whiteboard module specifically handles vector graphics actions, while scene renderers manage video playback and speech synchronization.

## Practical Code Examples for Defining Actions

Developers construct action objects following the TypeScript interfaces defined in the DSL package. These objects feed into scene generation pipelines or direct agent outputs.

Creating a speech action for AI teacher narration:

```typescript
// Create a speech action – the AI teacher will narrate this line
const speech: Action = {
  id: 'a1',
  type: 'speech',
  text: 'Welcome to the lesson on quantum mechanics!',
};

```

Embedding synchronized video content:

```typescript
// Play a video clip inside a slide
const video: Action = {
  id: 'a2',
  type: 'play_video',
  elementId: 'video_01',          // matches the video element in the slide
  startMs: 0,
  durationMs: 15000,
};

```

Rendering whiteboard diagrams with stroke and shape primitives:

```typescript
// Draw a whiteboard diagram (line + shape)
const draw: Action = {
  id: 'a3',
  type: 'draw_line',
  from: { x: 100, y: 200 },
  to:   { x: 300, y: 200 },
  color: '#0066ff',
  width: 3,
};

const shape: Action = {
  id: 'a4',
  type: 'draw_rect',
  rect: { x: 150, y: 250, width: 200, height: 120 },
  fill: '#f0f0f0',
  stroke: '#333333',
};

```

The runtime automatically serializes these objects to the client, where frontend components render the corresponding UI elements without manual intervention.

## Summary

- The OpenMAIC orchestration system supports **28+ action types** across nine functional categories, from audio synthesis to vector graphics rendering.
- Action definitions are centralized in **`packages/@openmaic/dsl/src/action.ts`**, providing type-safe discriminated unions for the entire system.
- Execution flows through a **LangGraph** orchestration layer (`lib/orchestration/`) to the **agent runtime** (`lib/server/agent-runtime/`) and **action executor** (`lib/action/`).
- Frontend components in `components/whiteboard/` and `components/scene-renderers/` handle the final rendering of whiteboard drawings, video playback, and speech synthesis.
- The system supports deterministic replay, cancellation, and steering of active sessions through its event-log-based architecture.

## Frequently Asked Questions

### How many action types does the OpenMAIC orchestration system support?

The system currently recognizes **28+ distinct action types**, including primitives for speech synthesis (`speech`, `tts`), video control (`play_video`, `seek_video`), whiteboard drawing (`draw_line`, `draw_rect`), and session management (`wait`, `cancel`). These are defined as a TypeScript discriminated union in `packages/@openmaic/dsl/src/action.ts`.

### Where are action handlers implemented in the OpenMAIC codebase?

Concrete execution logic resides in the **`lib/action/`** directory, which contains specialized handlers for each action category. The speech handler manages TTS audio streams, while whiteboard handlers generate SVG primitives. The orchestration graph (`lib/orchestration/`) routes these actions, and the agent runtime (`lib/server/agent-runtime/`) manages their durable execution.

### Can the OpenMAIC orchestration system resume or cancel in-progress actions?

Yes. The LangGraph-based **orchestration layer** maintains a deterministic execution graph and event log that enables the system to **resume, cancel, or steer** running sessions. Control flow actions like `wait`, `delay`, and `cancel` provide explicit mechanisms for managing execution state, while the underlying runtime preserves session continuity through its durable event architecture.

### How are whiteboard drawing actions rendered on the client?

Whiteboard actions (such as `draw_line`, `draw_rect`, and `draw_path`) are serialized from the action executor to the client, where the **`components/whiteboard/`** module deserializes them and renders corresponding SVG primitives. Each action carries geometric parameters (coordinates, colors, stroke widths) that the frontend translates into immediate canvas updates, supporting real-time collaborative drawing during AI-led sessions.