What Types of Actions Are Supported by the OpenMAIC Orchestration System?

The OpenMAIC orchestration system supports 28+ distinct action types—including speech synthesis, video playback, whiteboard drawing, and session control—defined centrally in packages/@openmaic/dsl/src/action.ts and executed through a LangGraph-based action engine.

The THU-MAIC/OpenMAIC repository implements a sophisticated orchestration layer that manages AI-driven educational sessions through a deterministic Action Engine. This system recognizes diverse action types that enable agents to manipulate multimedia content, render interactive whiteboard primitives, and control application flow in real time.

Action Types Supported by the OpenMAIC Orchestration System

The Action Engine categorizes its supported operations into nine functional groups. These definitions reside in packages/@openmaic/dsl/src/action.ts, which serves as the central TypeScript discriminant union for all possible operations.

Speech and Audio Actions

Audio narration capabilities include:

• speech – Initiates agent dialogue with natural language text
• tts – Direct text-to-speech synthesis requests
• play_audio – Streams pre-recorded audio assets to the client

Video Control Actions

The system manages embedded video content through:

• play_video – Starts playback from a specified millisecond offset
• pause_video – Suspends active video streams
• seek_video – Repositions playback to a specific timestamp

Whiteboard Drawing Actions

Vector graphics rendering supports seven distinct primitives:

• draw_line – Renders linear strokes between coordinates
• draw_rect – Creates rectangular shapes with fill and stroke properties
• draw_ellipse – Generates elliptical forms
• draw_path – Executes complex SVG path commands
• draw_text – Places typographic elements on the canvas
• draw_shape – Generic shape instantiation
• draw_chart – Data visualization rendering

Spotlight and Laser Actions

Presentation aids include:

• spotlight – Focuses user attention on specific screen regions
• laser_pointer – Simulates laser pointer movement across content

Material Handling Actions

Content management operations:

• material_extraction – Retrieves learning materials from storage
• material_upload – Persists user-generated content
• material_download – Delivers resources to local client storage

Generation Helper Actions

Meta-operations for dynamic scene construction:

• generate_actions – Creates action sequences programmatically
• edit_actions – Modifies existing action chains
• generate_scene – Constructs complete scene definitions
• edit_scene – Updates active scene configurations

Control and Flow Actions

Session state management primitives:

• wait – Blocks execution until conditions are met
• delay – Introduces timed pauses
• cancel – Terminates pending operations
• reconnect – Restores dropped client connections

UI Manipulation Actions

Interface state transitions:

• expand_folder – Expands hierarchical navigation nodes
• collapse_folder – Collapses directory structures
• select_tab – Activates specific interface tabs
• open_course – Loads course-level content containers

Miscellaneous Actions

Utility operations including highlight, show_hint, and raise_event for custom signaling.

Action Execution Pipeline and Architecture

The OpenMAIC orchestration system processes these action types through a deterministic pipeline that ensures consistent execution and replayability.

Definition Layer

The DSL package (packages/@openmaic/dsl/src/action.ts) defines the discriminated union of all 28+ action kinds using TypeScript's type system. Each action carries a unique id and type property that routes it through the system.

Orchestration Graph

The LangGraph-based orchestration layer (lib/orchestration/) routes agent turns and schedules action dispatch. This graph maintains session state and determines the execution order of operations, allowing the system to resume, cancel, or steer running sessions while preserving an event log.

Runtime and Execution

The agent runtime (lib/server/agent-runtime/) consumes action definitions and manages durable session execution. Actions pass to the action executor (lib/action/), which contains concrete handlers for each type. For example, the speech handler streams TTS audio to the client, while whiteboard handlers render SVG primitives.

Frontend Rendering

Client-side components in components/whiteboard/ and components/scene-renderers/ deserialize actions and update the UI. The whiteboard module specifically handles vector graphics actions, while scene renderers manage video playback and speech synchronization.

Practical Code Examples for Defining Actions

Developers construct action objects following the TypeScript interfaces defined in the DSL package. These objects feed into scene generation pipelines or direct agent outputs.

Creating a speech action for AI teacher narration:

// Create a speech action – the AI teacher will narrate this line
const speech: Action = {
  id: 'a1',
  type: 'speech',
  text: 'Welcome to the lesson on quantum mechanics!',
};

Embedding synchronized video content:

// Play a video clip inside a slide
const video: Action = {
  id: 'a2',
  type: 'play_video',
  elementId: 'video_01',          // matches the video element in the slide
  startMs: 0,
  durationMs: 15000,
};

Rendering whiteboard diagrams with stroke and shape primitives:

// Draw a whiteboard diagram (line + shape)
const draw: Action = {
  id: 'a3',
  type: 'draw_line',
  from: { x: 100, y: 200 },
  to:   { x: 300, y: 200 },
  color: '#0066ff',
  width: 3,
};

const shape: Action = {
  id: 'a4',
  type: 'draw_rect',
  rect: { x: 150, y: 250, width: 200, height: 120 },
  fill: '#f0f0f0',
  stroke: '#333333',
};

The runtime automatically serializes these objects to the client, where frontend components render the corresponding UI elements without manual intervention.

Summary

• The OpenMAIC orchestration system supports 28+ action types across nine functional categories, from audio synthesis to vector graphics rendering.
• Action definitions are centralized in packages/@openmaic/dsl/src/action.ts, providing type-safe discriminated unions for the entire system.
• Execution flows through a LangGraph orchestration layer (lib/orchestration/) to the agent runtime (lib/server/agent-runtime/) and action executor (lib/action/).
• Frontend components in components/whiteboard/ and components/scene-renderers/ handle the final rendering of whiteboard drawings, video playback, and speech synthesis.
• The system supports deterministic replay, cancellation, and steering of active sessions through its event-log-based architecture.

Frequently Asked Questions

How many action types does the OpenMAIC orchestration system support?

The system currently recognizes 28+ distinct action types, including primitives for speech synthesis (speech, tts), video control (play_video, seek_video), whiteboard drawing (draw_line, draw_rect), and session management (wait, cancel). These are defined as a TypeScript discriminated union in packages/@openmaic/dsl/src/action.ts.

Where are action handlers implemented in the OpenMAIC codebase?

Concrete execution logic resides in the lib/action/ directory, which contains specialized handlers for each action category. The speech handler manages TTS audio streams, while whiteboard handlers generate SVG primitives. The orchestration graph (lib/orchestration/) routes these actions, and the agent runtime (lib/server/agent-runtime/) manages their durable execution.

Can the OpenMAIC orchestration system resume or cancel in-progress actions?

Yes. The LangGraph-based orchestration layer maintains a deterministic execution graph and event log that enables the system to resume, cancel, or steer running sessions. Control flow actions like wait, delay, and cancel provide explicit mechanisms for managing execution state, while the underlying runtime preserves session continuity through its durable event architecture.

How are whiteboard drawing actions rendered on the client?

Whiteboard actions (such as draw_line, draw_rect, and draw_path) are serialized from the action executor to the client, where the components/whiteboard/ module deserializes them and renders corresponding SVG primitives. Each action carries geometric parameters (coordinates, colors, stroke widths) that the frontend translates into immediate canvas updates, supporting real-time collaborative drawing during AI-led sessions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}