Complete Guide to OpenMAIC Action Engine Types: Fire-and-Forget vs Synchronous Actions

The OpenMAIC action engine supports over 20 distinct action types defined in packages/@openmaic/dsl/src/action.ts, categorized into non-blocking fire-and-forget visual effects (spotlight, laser) and synchronous blocking operations (speech, video playback, whiteboard drawing, widget interactions, and discussions).

The OpenMAIC action engine serves as the execution backbone for the THU-MAIC/OpenMAIC repository, driving interactive presentations through a declarative Domain Specific Language (DSL). Every operation—from highlighting slide elements to triggering complex widget simulations—centers around a single Action contract that extends ActionBase with id, title, and description fields.

Core Architecture of the OpenMAIC Action Engine

All action implementations derive from ActionBase in packages/@openmaic/dsl/src/action.ts. The runtime distinguishes between two execution semantics:

• Fire-and-Forget: Instant visual effects that execute without blocking the action pipeline
• Synchronous: Blocking operations that pause the engine until completion (TTS, animations, user interactions)

Concrete types are collected into the union type Action (lines 335–356) and the ACTION_TYPES array (lines 89–112) for runtime membership validation. Additional runtime logic resides in lib/playback/action-navigation.ts and lib/playback/action-resume.ts, while lib/chat/pi/tools/classroom-actions.ts integrates these actions into the chat-based classroom assistant.

Fire-and-Forget Action Types

These actions provide immediate visual feedback without interrupting the sequence flow.

Spotlight

The spotlight action focuses attention on a single slide element while dimming surrounding content. Defined at lines 30–34 in packages/@openmaic/dsl/src/action.ts, it accepts elementId and dimOpacity parameters.

import { Action } from '@openmaic/dsl';

const spotlight: Action = {
  id: 'a1',
  type: 'spotlight',
  elementId: 'logo',
  dimOpacity: 0.4,
};

Laser

The laser action renders a laser pointer-style line on slide elements (lines 37–41). Like spotlight, it fires instantly and releases control to the next action.

Synchronous Action Types

These actions block the OpenMAIC action engine until completion, ensuring sequential dependencies resolve correctly.

Speech

The speech action triggers teacher narration, blocking until Text-to-Speech (TTS) or provided audio finishes. Located at lines 46–52 in packages/@openmaic/dsl/src/action.ts, it supports text, voice, and speed parameters.

import { Action } from '@openmaic/dsl';

const narrate: Action = {
  id: 'a2',
  type: 'speech',
  text: 'Welcome to the lesson on neural networks.',
  voice: 'en-US-Wavenet-D',
  speed: 1.0,
};

Video Playback

The play_video action initiates slide-embedded video playback (lines 89–93), blocking until playback completes or is manually terminated.

Whiteboard Operations (wb_*)

The engine provides nine whiteboard-specific actions for canvas manipulation, all defined in packages/@openmaic/dsl/src/action.ts:

• wb_open (lines 64–66): Opens the whiteboard canvas with animation awareness
• wb_draw_text (lines 69–78): Renders text blocks at specified x/y coordinates
• wb_draw_shape (lines 82–90): Draws geometric primitives (rectangles, circles, triangles)
• wb_draw_chart (lines 94–100): Renders data visualizations (bar, line, pie charts) with supplied datasets
• wb_draw_latex (lines 111–118): Displays mathematical formulas
• wb_draw_table (lines 123–132): Creates tables from 2D string arrays
• wb_draw_line (lines 136–146): Draws connecting lines or arrows between points
• wb_clear (lines 150–152): Removes all canvas elements
• wb_delete (lines 155–158): Removes a specific element by its elementId
• wb_close (lines 160–164): Closes the whiteboard with animation support

const lessonActions: Action[] = [
  { id: 'a3', type: 'wb_open' },
  {
    id: 'a4',
    type: 'wb_draw_text',
    content: 'Hello, world!',
    x: 100,
    y: 150,
  },
  { id: 'a5', type: 'wb_draw_chart', chartType: 'bar', data: [10, 20, 30] },
  { id: 'a6', type: 'wb_close' },
];

Widget Interactions (widget_*)

For embedded iframe widgets, the OpenMAIC action engine supports four manipulation actions defined in the DSL:

• widget_highlight (lines 204–209): Highlights elements inside widget iframes
• widget_setState (lines 212–217): Updates widget internal state (e.g., simulation variables)
• widget_annotation (lines 220–224): Adds floating annotations to widget elements
• widget_reveal (lines 226–231): Exposes hidden content within widgets

const setGravity: Action = {
  id: 'a5',
  type: 'widget_setState',
  state: { gravity: 9.81 },
  content: 'Adjusting gravity to Earth standard.',
};

Discussion

The discussion action triggers round-table discussions with optional topic and prompt parameters (lines 194–200), blocking until the discussion phase concludes.

Runtime Integration and Key Files

Beyond type definitions, several files implement the OpenMAIC action engine behavior:

• packages/@openmaic/dsl/src/action.ts: Central type definitions and the Action union (lines 335–356)
• lib/types/action.ts: Legacy re-export shim maintaining backward compatibility
• lib/playback/action-navigation.ts: Runtime interpretation and execution logic during playback
• lib/playback/action-resume.ts: State management for paused action sequences
• components/workbench/chat/action-cluster.tsx: UI component grouping actions in the workbench interface

Summary

• The OpenMAIC action engine processes 20+ distinct action types defined in packages/@openmaic/dsl/src/action.ts
• Fire-and-forget actions (spotlight, laser) execute instantly without blocking subsequent operations
• Synchronous actions (speech, play_video, wb_*, widget_*, discussion) pause the engine until completion
• Whiteboard operations support complete canvas lifecycles from wb_open to wb_close with drawing primitives for text, shapes, charts, LaTeX, tables, and lines
• Widget actions enable state manipulation and annotation of embedded iframe content
• All action types extend ActionBase and compile into the Action union type (lines 335–356) for type-safe runtime validation

Frequently Asked Questions

What is the difference between fire-and-forget and synchronous actions in OpenMAIC?

Fire-and-forget actions like spotlight and laser execute visual effects instantly without blocking the action pipeline, allowing the engine to proceed immediately to the next instruction. Synchronous actions such as speech, play_video, and all wb_* operations pause execution until the current action completes, ensuring that narrations finish or animations play out before subsequent actions trigger.

How do I implement a complete whiteboard sequence using OpenMAIC action types?

A complete whiteboard workflow requires at minimum wb_open to initialize the canvas and wb_close to terminate it, with drawing actions in between. Valid intermediate actions include wb_draw_text for annotations, wb_draw_shape for geometry, wb_draw_chart for data visualization, and wb_clear or wb_delete for element management, all defined between lines 64–164 in packages/@openmaic/dsl/src/action.ts.

Can OpenMAIC actions manipulate third-party widget content?

Yes, the widget_* action family provides four specific interaction types: widget_highlight for element focus, widget_setState for updating simulation variables, widget_annotation for floating labels, and widget_reveal for content disclosure. These actions target embedded iframe widgets and are defined at lines 204–231 in the DSL package.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}