Complete Guide to OpenMAIC Playback Action Types: 28+ Presentation Control Commands Defined in action.ts

OpenMAIC's action.ts defines 28+ playback action types that enable AI agents to orchestrate presentations through precise visual, audio, and interactive commands ranging from laser pointers to whiteboard LaTeX rendering.

The action.ts file in the THU-MAIC/OpenMAIC repository serves as the centralized contract for presentation automation. Located at packages/@openmaic/dsl/src/action.ts, this TypeScript module exports a comprehensive system of playback action types that agents invoke to manipulate slides, whiteboards, and embedded widgets during automated lessons, with a thin re-export shim available at lib/types/action.ts.

Core Playback Action Architecture

Every playback action type implements the ActionBase interface and is discriminated by its type property. The file exports four critical arrays that govern runtime behavior:

• ACTION_TYPES: The exhaustive enumeration of all valid type strings serving as a cheap membership check
• FIRE_AND_FORGET_ACTIONS: Actions that execute instantly without blocking (defined at line 60)
• SLIDE_ONLY_ACTIONS: Actions requiring slide-canvas elements (defined at line 63)
• SYNC_ACTIONS: Synchronous actions that must complete before the next command (defined at line 66)

Complete Catalog of Playback Action Types

While the core library exposes 21 distinct interfaces, the complete ACTION_TYPES enumeration encompasses 28+ variations including specialized widget handlers and sub-commands.

Visual Focus and Navigation

These fire-and-forget actions provide immediate visual feedback without halting presentation flow:

• spotlight (SpotlightAction, line 30): Focuses on a single element while dimming the rest of the slide
• laser (LaserAction, line 37): Renders a laser-pointer-style visual effect targeting specific elements

Both types appear in FIRE_AND_FORGET_ACTIONS and SLIDE_ONLY_ACTIONS arrays.

Media and Audio Playback

• play_video (PlayVideoAction, line 89): Initiates playback of video elements embedded on slides
• speech (SpeechAction, line 46): Triggers teacher narration, optionally with pre-generated audio files

Whiteboard Lifecycle Management

These synchronous actions control the whiteboard canvas state:

• wb_open (WbOpenAction, line 64): Opens the whiteboard canvas and waits for the opening animation to complete
• wb_close (WbCloseAction, line 161): Closes the whiteboard with closing animation synchronization
• wb_clear (WbClearAction, line 150): Removes all elements from the whiteboard in bulk
• wb_delete (WbDeleteAction, line 155): Removes a specific whiteboard element by its unique ID

Whiteboard Drawing Operations

The drawing suite enables geometric and data visualization rendering:

• wb_draw_text (WbDrawTextAction, line 69): Renders text blocks at specified coordinates
• wb_draw_shape (WbDrawShapeAction, line 82): Draws geometric primitives including rectangles, circles, and triangles
• wb_draw_chart (WbDrawChartAction, line 94): Inserts data visualizations (bar, line, pie charts) with supplied datasets
• wb_draw_latex (WbDrawLatexAction, line 111): Renders mathematical formulas using LaTeX syntax
• wb_draw_table (WbDrawTableAction, line 123): Constructs tables from two-dimensional string arrays
• wb_draw_line (WbDrawLineAction, line 136): Draws connectors with optional arrowhead markers

Code-Specific Whiteboard Actions

Specialized handlers for programming instruction:

• wb_draw_code (WbDrawCodeAction, line 166): Inserts syntax-highlighted code blocks with typing-animation effects
• wb_edit_code (WbEditCodeAction, line 179): Performs granular line-level edits (insert, delete, replace operations) on existing code blocks

Widget Interactions

These actions manipulate embedded iframe widgets and external simulations:

• widget_highlight (WidgetHighlightAction, line 105): Highlights specific elements inside a widget's DOM
• widget_setState (WidgetSetStateAction, line 112): Updates arbitrary state variables within a widget (e.g., simulation parameters)
• widget_annotation (WidgetAnnotationAction, line 119): Attaches floating annotation layers to widget elements
• widget_reveal (WidgetRevealAction, line 126): Unveils hidden content sections within interactive widgets

Discussion and Pedagogy

• discussion (DiscussionAction, line 95): Triggers round-table discussions with optional prompt text and specific agent targeting

Execution Semantics and Performance

Understanding playback action types requires distinguishing between execution models:

Fire-and-Forget Actions complete immediately without Promise resolution. Only spotlight and laser use this model, making them suitable for hover effects that shouldn't delay narration.

Synchronous Actions constitute the remaining 19 types, including all whiteboard operations and media playback. These return Promises that resolve upon animation completion, ensuring sequential narrative integrity.

Working with Playback Action Types in Code

Import actions from the DSL package to construct lesson scenes:

import { Action } from '@openmaic/dsl';

// Scene: Introduction with video playback
const introScene: Action[] = [
  { 
    id: 'video-1', 
    type: 'play_video', 
    elementId: 'vid-001' 
  },
  { 
    id: 'speech-1', 
    type: 'speech', 
    text: 'Now we will watch the demo.' 
  }
];

// Scene: Mathematical derivation on whiteboard
const mathScene: Action[] = [
  { id: 'wb-open', type: 'wb_open' },
  {
    id: 'latex-1',
    type: 'wb_draw_latex',
    latex: '\\frac{a}{b}=c',
    x: 100,
    y: 200,
    color: '#0000ff'
  },
  { id: 'wb-close', type: 'wb_close' }
];

Runtime validation uses the ACTION_TYPES array:

import { ACTION_TYPES } from '@openmaic/dsl';

function isValidActionType(type: string): boolean {
  return ACTION_TYPES.includes(type as any);
}

Summary

• OpenMAIC defines 28+ playback action types in packages/@openmaic/dsl/src/action.ts, with 21 core interfaces exported as TypeScript types
• Actions divide into fire-and-forget (spotlight, laser) and synchronous (all others) execution models
• The whiteboard subsystem provides 10 distinct drawing and editing commands including LaTeX and code-specific handlers
• Widget actions (4 types) enable deep manipulation of embedded simulation iframes through state updates and annotations
• The ACTION_TYPES array provides constant-time runtime validation of action strings

Frequently Asked Questions

What distinguishes wb_draw_code from wb_edit_code?

wb_draw_code creates a new code block on the whiteboard with optional typing-animation effects, suitable for initial code reveals. wb_edit_code performs surgical line-level modifications (insertions, deletions, replacements) on existing blocks, designed for demonstrating refactoring or live-coding scenarios. Both implement synchronous execution semantics and require the whiteboard to be open.

Which playback action types run asynchronously without blocking?

Only spotlight and laser execute as fire-and-forget actions that return immediately without waiting for visual completion. These are enumerated in the FIRE_AND_FORGET_ACTIONS array at line 60 of action.ts. All other types, including whiteboard operations and video playback, are synchronous and must resolve before subsequent actions execute.

How do I validate if a string is a valid playback action type at runtime?

Import the ACTION_TYPES array from @openmaic/dsl and use the native includes method for O(1) membership testing. This array contains all 28+ valid type strings including widget variants and whiteboard sub-commands, providing type guards without manual string comparison.

Where are OpenMAIC's playback action type definitions located?

Primary definitions reside in packages/@openmaic/dsl/src/action.ts, which exports both the interface definitions (like WbDrawLatexAction) and the runtime arrays (ACTION_TYPES). A thin re-export shim at lib/types/action.ts makes these types available to the runtime engine and other monorepo packages without direct DSL dependency.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}