What Actions Can the OpenMAIC Action Engine Execute?

The OpenMAIC action engine can execute 18 distinct action types, including text-to-speech synthesis, video playback control, whiteboard drawing and editing, slide navigation, tool invocations, material upload and extraction, session lifecycle management, and error handling.

This guide provides the complete technical reference for the Open Multi-Agent Interactive Classroom (OpenMAIC) action engine. All actions are defined in the OpenMAIC DSL (@openmaic/dsl) as a discriminated-union type Action, with the engine dispatching each to its appropriate runtime handler based on the type field.

---

Speech and Narration Actions

The OpenMAIC action engine generates spoken narration through text-to-speech synthesis.

Action	Description	Key Parameters
speech	Text-to-speech synthesis for AI narration or assistant voice	id, text, voice?, audioUrl?

In packages/@openmaic/dsl/src/action.ts, the speech action type enables dynamic voice generation. The engine routes this to a TTS provider through the generate_scene_actions dispatcher in lib/server/agent-runtime/generation-tools.ts.

import { Action } from '@openmaic/dsl';

const speak: Action = {
  type: 'speech',
  id: 'a-001',
  text: 'Welcome to the quantum physics lesson!',
  voice: 'en-US-Standard-B',
};

await generateSceneActions([speak]);

The optional audioUrl parameter allows pre-generated audio to be substituted for on-the-fly synthesis, improving latency for repeated content.

---

Video Playback Actions

The action engine controls embedded video elements through three discrete action types.

Action	Description
play_video	Starts playback, optionally seeking to a timestamp
pause_video	Pauses a playing video
stop_video	Stops playback and resets to the start frame

const play: Action = {
  type: 'play_video',
  elementId: 'video_42',
  seekMs: 0,
};

await generateSceneActions([play]);

All video actions target specific elements via elementId, defined in the slide model. The runtime maps these to HTML5 video player controls.

---

Whiteboard Drawing and Annotation Actions

The whiteboard subsystem supports six action types for interactive annotation, as implemented in the canvas renderer.

Basic Drawing Actions

Action	Description
wb_draw_line	Draws straight lines between two points
wb_draw_text	Renders text blocks with font and color control
wb_draw_shape	Draws rectangles, ellipses, and polygons

const line: Action = {
  type: 'wb_draw_line',
  id: 'w-01',
  from: { x: 100, y: 150 },
  to:   { x: 400, y: 150 },
  color: '#ff5722',
  strokeWidth: 3,
};

await generateSceneActions([line]);

Interactive Manipulation Actions

Action	Description
wb_erase	Removes content by region or object ID
wb_highlight	Temporary visual emphasis with duration
wb_focus	Camera movement to center on target with optional zoom

The wb_highlight action creates transient attention cues, while wb_focus enables programmatic camera control for guided instruction.

---

Navigation Actions

The action engine manages spatial and structural navigation through two action types.

Action	Description
navigate_slide	Switches to specific slide or directional movement
navigate_course	Jumps between courses or modules

const nav: Action = {
  type: 'navigate_slide',
  slideId: 'slide-07',
  direction: 'next'  // or 'prev'
};

These actions update the presentation state observable by all connected clients, maintaining synchronization across the collaborative classroom.

---

Tool Invocation and Generation Actions

The generate_actions type enables the engine to call external skills and tools, making the system extensible.

Action	Description
generate_actions	Invokes a tool that returns new actions (e.g., slide generation, quiz creation)
tool_progress	Emits progress updates for long-running operations
tool_error	Signals tool failure with diagnostic information

const gen: Action = {
  type: 'generate_actions',
  toolName: 'slide_generator',
  toolDetails: { order: 1 },
  input: { topic: 'Newtonian Mechanics' },
};

await generateSceneActions([gen]);  // tool returns further actions for execution

The tool runtime in lib/server/agent-runtime/generation-tools.ts handles the request-response cycle, streaming progress via tool_progress actions and capturing errors through tool_error.

---

Action Modification Actions

The edit_actions type implements JSON-Patch-style mutation of existing actions.

Action	Description
edit_actions	Modifies an existing action's properties

const edit: Action = {
  type: 'edit_actions',
  targetActionId: 'a-001',
  patch: [{ op: 'replace', path: '/text', value: 'Let’s dive deeper.' }],
};

await generateSceneActions([edit]);

This pattern supports collaborative editing and AI-driven refinement, where generated content is subsequently revised without full regeneration. Test coverage appears in tests/workbench/tool-presentation.test.ts.

---

Material Management Actions

The engine handles learning content ingestion through three material-focused action types, as implemented in lib/workbench/material-upload-scheduling.ts and related modules.

Action	Description
material_upload	Uploads PDF, audio, or video for later use
material_extraction	Initiates background processing (OCR, transcription)
edit_material	Updates metadata or replacement content

The material_extraction action carries a status field with states: running, finished, or failed. Extraction flows are tested in tests/workbench/session-fold.test.ts.

---

Session Lifecycle and Resilience Actions

The action engine maintains durable workspace sessions with explicit lifecycle management, stored in lib/server/agent-runtime/store.ts.

Action	Description
session_start	Initializes a new durable session
session_end	Gracefully terminates a session
session_cancel	Aborts an ongoing session
reconnect	Re-establishes WebSocket after transient disconnect

const reconnect: Action = {
  type: 'reconnect',
  sessionId: 'sess-uuid-1234'
};

These actions enable resumable and steerable generation—users can cancel, pause, or resume AI-driven content creation. The session API is defined in lib/workbench/workspace-actions.ts with test coverage in tests/workbench/owner-session-client.test.ts.

---

How the OpenMAIC Action Engine Processes Actions

Understanding the execution pipeline clarifies how these 18 action types operate:

1. Parsing — Incoming objects are validated against the Action union type in packages/@openmaic/dsl/src/action.ts

2. Dispatch — The generate_scene_actions function routes each action to its runtime: TTS service, HTML5 video player, whiteboard canvas, or skill backend

3. Execution — Side effects occur (API calls, state updates, rendering), producing result snapshots

4. Persistence — Session actions are written to the agent-session store, enabling recovery and replay

---

Summary

• 18 action types are available in the OpenMAIC action engine, defined in packages/@openmaic/dsl/src/action.ts
• Speech and video actions handle multimedia presentation
• Whiteboard actions (wb_*) provide six primitives for interactive annotation
• Navigation actions control slide and course structure
• Tool actions (generate_actions, tool_progress, tool_error) enable extensible AI skill integration
• Modification actions allow runtime patching of existing actions
• Material actions manage content ingestion and background processing
• Session actions provide durability, cancellation, and reconnect resilience

---

Frequently Asked Questions

What is the OpenMAIC action engine?

The OpenMAIC action engine is a runtime dispatcher that executes 18 distinct action types defined in the @openmaic/dsl package. It parses incoming action objects, routes them to appropriate handlers (TTS, video, whiteboard, tools), executes side effects, and persists session state for resumability.

How are new action types added to OpenMAIC?

New action types require three changes: extend the Action union type in packages/@openmaic/dsl/src/action.ts, add a handler branch in lib/server/agent-runtime/generation-tools.ts, and implement the corresponding runtime module. The discriminated-union pattern with type field dispatch makes extension straightforward.

Can OpenMAIC actions be modified after creation?

Yes, the edit_actions type supports JSON-Patch-style modifications to existing actions. Specify the targetActionId and an RFC 6902 compliant patch array to replace, add, or remove properties without regenerating the entire action sequence.

How does OpenMAIC handle long-running tool execution?

The engine uses tool_progress actions to stream percentage and stage updates to clients, and tool_error to capture failures with structured diagnostic codes. Sessions can be cancelled mid-execution via session_cancel, with state preserved in lib/server/agent-runtime/store.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}