# Complete Guide to OpenMAIC Playback Action Types: 28+ Presentation Control Commands Defined in action.ts

> Explore over 28 OpenMAIC playback action types in action.ts for AI presentation control. Master commands for visuals, audio, and interactivity.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: api-reference
- Published: 2026-09-04

---

**OpenMAIC's [`action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/action.ts) defines 28+ playback action types** that enable AI agents to orchestrate presentations through precise visual, audio, and interactive commands ranging from laser pointers to whiteboard LaTeX rendering.

The [`action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/action.ts) file in the [THU-MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC) repository serves as the centralized contract for presentation automation. Located at `packages/@openmaic/dsl/src/action.ts`, this TypeScript module exports a comprehensive system of **playback action types** that agents invoke to manipulate slides, whiteboards, and embedded widgets during automated lessons, with a thin re-export shim available at [`lib/types/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts).

## Core Playback Action Architecture

Every **playback action type** implements the `ActionBase` interface and is discriminated by its `type` property. The file exports four critical arrays that govern runtime behavior:

- **`ACTION_TYPES`**: The exhaustive enumeration of all valid type strings serving as a cheap membership check
- **`FIRE_AND_FORGET_ACTIONS`**: Actions that execute instantly without blocking (defined at line 60)
- **`SLIDE_ONLY_ACTIONS`**: Actions requiring slide-canvas elements (defined at line 63)
- **`SYNC_ACTIONS`**: Synchronous actions that must complete before the next command (defined at line 66)

## Complete Catalog of Playback Action Types

While the core library exposes 21 distinct interfaces, the complete `ACTION_TYPES` enumeration encompasses 28+ variations including specialized widget handlers and sub-commands.

### Visual Focus and Navigation

These **fire-and-forget** actions provide immediate visual feedback without halting presentation flow:

- **`spotlight`** (`SpotlightAction`, line 30): Focuses on a single element while dimming the rest of the slide
- **`laser`** (`LaserAction`, line 37): Renders a laser-pointer-style visual effect targeting specific elements

Both types appear in `FIRE_AND_FORGET_ACTIONS` and `SLIDE_ONLY_ACTIONS` arrays.

### Media and Audio Playback

- **`play_video`** (`PlayVideoAction`, line 89): Initiates playback of video elements embedded on slides
- **`speech`** (`SpeechAction`, line 46): Triggers teacher narration, optionally with pre-generated audio files

### Whiteboard Lifecycle Management

These synchronous actions control the whiteboard canvas state:

- **`wb_open`** (`WbOpenAction`, line 64): Opens the whiteboard canvas and waits for the opening animation to complete
- **`wb_close`** (`WbCloseAction`, line 161): Closes the whiteboard with closing animation synchronization
- **`wb_clear`** (`WbClearAction`, line 150): Removes **all** elements from the whiteboard in bulk
- **`wb_delete`** (`WbDeleteAction`, line 155): Removes a specific whiteboard element by its unique ID

### Whiteboard Drawing Operations

The drawing suite enables geometric and data visualization rendering:

- **`wb_draw_text`** (`WbDrawTextAction`, line 69): Renders text blocks at specified coordinates
- **`wb_draw_shape`** (`WbDrawShapeAction`, line 82): Draws geometric primitives including rectangles, circles, and triangles
- **`wb_draw_chart`** (`WbDrawChartAction`, line 94): Inserts data visualizations (bar, line, pie charts) with supplied datasets
- **`wb_draw_latex`** (`WbDrawLatexAction`, line 111): Renders mathematical formulas using LaTeX syntax
- **`wb_draw_table`** (`WbDrawTableAction`, line 123): Constructs tables from two-dimensional string arrays
- **`wb_draw_line`** (`WbDrawLineAction`, line 136): Draws connectors with optional arrowhead markers

### Code-Specific Whiteboard Actions

Specialized handlers for programming instruction:

- **`wb_draw_code`** (`WbDrawCodeAction`, line 166): Inserts syntax-highlighted code blocks with typing-animation effects
- **`wb_edit_code`** (`WbEditCodeAction`, line 179): Performs granular line-level edits (insert, delete, replace operations) on existing code blocks

### Widget Interactions

These actions manipulate embedded iframe widgets and external simulations:

- **`widget_highlight`** (`WidgetHighlightAction`, line 105): Highlights specific elements inside a widget's DOM
- **`widget_setState`** (`WidgetSetStateAction`, line 112): Updates arbitrary state variables within a widget (e.g., simulation parameters)
- **`widget_annotation`** (`WidgetAnnotationAction`, line 119): Attaches floating annotation layers to widget elements
- **`widget_reveal`** (`WidgetRevealAction`, line 126): Unveils hidden content sections within interactive widgets

### Discussion and Pedagogy

- **`discussion`** (`DiscussionAction`, line 95): Triggers round-table discussions with optional prompt text and specific agent targeting

## Execution Semantics and Performance

Understanding **playback action types** requires distinguishing between execution models:

**Fire-and-Forget Actions** complete immediately without Promise resolution. Only `spotlight` and `laser` use this model, making them suitable for hover effects that shouldn't delay narration.

**Synchronous Actions** constitute the remaining 19 types, including all whiteboard operations and media playback. These return Promises that resolve upon animation completion, ensuring sequential narrative integrity.

## Working with Playback Action Types in Code

Import actions from the DSL package to construct lesson scenes:

```typescript
import { Action } from '@openmaic/dsl';

// Scene: Introduction with video playback
const introScene: Action[] = [
  { 
    id: 'video-1', 
    type: 'play_video', 
    elementId: 'vid-001' 
  },
  { 
    id: 'speech-1', 
    type: 'speech', 
    text: 'Now we will watch the demo.' 
  }
];

// Scene: Mathematical derivation on whiteboard
const mathScene: Action[] = [
  { id: 'wb-open', type: 'wb_open' },
  {
    id: 'latex-1',
    type: 'wb_draw_latex',
    latex: '\\frac{a}{b}=c',
    x: 100,
    y: 200,
    color: '#0000ff'
  },
  { id: 'wb-close', type: 'wb_close' }
];

```

Runtime validation uses the `ACTION_TYPES` array:

```typescript
import { ACTION_TYPES } from '@openmaic/dsl';

function isValidActionType(type: string): boolean {
  return ACTION_TYPES.includes(type as any);
}

```

## Summary

- **OpenMAIC** defines 28+ **playback action types** in `packages/@openmaic/dsl/src/action.ts`, with 21 core interfaces exported as TypeScript types
- Actions divide into **fire-and-forget** (`spotlight`, `laser`) and **synchronous** (all others) execution models
- The **whiteboard** subsystem provides 10 distinct drawing and editing commands including LaTeX and code-specific handlers
- **Widget actions** (4 types) enable deep manipulation of embedded simulation iframes through state updates and annotations
- The `ACTION_TYPES` array provides constant-time runtime validation of action strings

## Frequently Asked Questions

### What distinguishes wb_draw_code from wb_edit_code?

**`wb_draw_code`** creates a new code block on the whiteboard with optional typing-animation effects, suitable for initial code reveals. **`wb_edit_code`** performs surgical line-level modifications (insertions, deletions, replacements) on existing blocks, designed for demonstrating refactoring or live-coding scenarios. Both implement synchronous execution semantics and require the whiteboard to be open.

### Which playback action types run asynchronously without blocking?

Only **`spotlight`** and **`laser`** execute as **fire-and-forget** actions that return immediately without waiting for visual completion. These are enumerated in the `FIRE_AND_FORGET_ACTIONS` array at line 60 of [`action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/action.ts). All other types, including whiteboard operations and video playback, are synchronous and must resolve before subsequent actions execute.

### How do I validate if a string is a valid playback action type at runtime?

Import the `ACTION_TYPES` array from `@openmaic/dsl` and use the native `includes` method for O(1) membership testing. This array contains all 28+ valid type strings including widget variants and whiteboard sub-commands, providing type guards without manual string comparison.

### Where are OpenMAIC's playback action type definitions located?

Primary definitions reside in **`packages/@openmaic/dsl/src/action.ts`**, which exports both the interface definitions (like `WbDrawLatexAction`) and the runtime arrays (`ACTION_TYPES`). A thin re-export shim at **[`lib/types/action.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts)** makes these types available to the runtime engine and other monorepo packages without direct DSL dependency.