OpenMAIC Classroom Interaction Action Types: Complete Guide to lib/action

The OpenMAIC platform supports 22 distinct classroom interaction action types through lib/action, organized into three execution categories: fire-and-forget visual effects, synchronous blocking operations, and widget interactions.

The OpenMAIC classroom platform defines a unified Action contract that drives every classroom interaction event—from spotlighting elements to playing speech narration to updating interactive widgets. All concrete action shapes are declared in the DSL package @openmaic/dsl and re-exported through lib/types/action.ts, ensuring every subsystem (runtime engine, renderer, importer, and UI) shares a single source of truth.

Action Type Categories in lib/action

The lib/action layer classifies classroom interaction action types by execution behavior. Understanding these categories is essential for lesson authors and platform developers working with the OpenMAIC source code.

Fire-and-Forget Actions (Non-Blocking Visual Effects)

These Action types execute independently without blocking subsequent actions. They terminate immediately after dispatch and are ideal for subtle visual cues.

Action Type	Interface	Source Location
spotlight	SpotlightAction	packages/@openmaic/dsl/src/action.ts L30-L34
laser	LaserAction	packages/@openmaic/dsl/src/action.ts L37-L41

Both types include elementId targeting and configurable dimOpacity for spotlight effects.

Synchronous Actions (Blocking Until Completion)

These Action types must complete before the lesson engine proceeds to the next action. They form the core of structured classroom interaction sequences.

Action Type	Purpose	Interface Location
speech	Text-to-speech narration with voice selection	L46-L62
wb_open	Initialize whiteboard session	L64-L66
wb_draw_text	Render text annotations	L69-L80
wb_draw_shape	Draw geometric primitives	L82-L92
wb_draw_chart	Render data visualizations	L94-L108
wb_draw_latex	Typeset mathematical expressions	L111-L121
wb_draw_table	Display tabular data	L124-L134
wb_draw_line	Animate connector lines	L136-L148
wb_clear	Erase whiteboard contents	L150-L153
wb_delete	Remove specific elements	L155-L159
wb_close	Terminate whiteboard session	L161-L164
wb_draw_code	Display syntax-highlighted code blocks	L166-L176
wb_edit_code	Enable live code editing	L179-L186
play_video	Embedded video playback	L188-L194
discussion	Facilitate student discussion prompts	L196-L200

Widget Interaction Actions

These specialized Action types manipulate embedded interactive widgets, enabling dynamic state changes and visual feedback.

Action Type	Function	Interface Location
widget_highlight	Emphasize widget boundaries	L204-L209
widget_setState	Update widget internal state	L212-L217
widget_annotation	Attach contextual notes	L219-L224
widget_reveal	Progressive disclosure of widget elements	L226-L231

Working with Action Types in Code

All Action interfaces are accessible through the runtime shim at lib/types/action.ts. This thin re-export preserves existing import patterns while centralizing the canonical definitions from @openmaic/dsl.

Import Pattern

import { Action, ActionType, SpotlightAction, SpeechAction } from '@/lib/types/action';

The Action union type encompasses all 22 concrete interfaces, while ActionType provides the string literal union for type discrimination.

Fire-and-Forget: Spotlight Action Example

import { SpotlightAction } from '@/lib/types/action';

const spotlight: SpotlightAction = {
  id: 'a1',
  type: 'spotlight',
  elementId: 'slide-3',
  dimOpacity: 0.4,
};

The id field provides unique identification for action tracking; elementId targets specific DOM elements; dimOpacity controls background dimming intensity (0.0-1.0).

Synchronous: Speech Action Example

import { SpeechAction } from '@/lib/types/action';

const speech: SpeechAction = {
  id: 'a2',
  type: 'speech',
  text: 'Welcome to the lesson on photosynthesis.',
  voice: 'en-US-Wavenet-D',
  rate: 1.0,
  pitch: 0,
};

The SpeechAction interface [L46-L62] supports Google Cloud TTS voices, playback rate modification, and pitch adjustment. The runtime engine at lib/playback/action-navigation.ts blocks until audio completion or user skip.

Whiteboard Drawing: Shape Action Example

import { WbDrawShapeAction } from '@/lib/types/action';

const shape: WbDrawShapeAction = {
  id: 'a3',
  type: 'wb_draw_shape',
  shape: 'rectangle',
  x: 150,
  y: 100,
  width: 300,
  height: 150,
  fillColor: '#5b9bd5',
  strokeColor: '#2e75b5',
  strokeWidth: 2,
};

Supported shape values include rectangle, ellipse, triangle, and arrow. The coordinate system uses absolute positioning relative to the whiteboard canvas origin.

Widget State Manipulation Example

import { WidgetSetStateAction } from '@/lib/types/action';

const setState: WidgetSetStateAction = {
  id: 'a4',
  type: 'widget_setState',
  widgetId: 'simulator-thermo',
  state: { temperature: 37, pressure: 101.3 },
  transition: 'smooth',
};

The state property accepts any JSON-serializable object matching the target widget's state schema. The optional transition field enables animated state interpolation.

Key Source Files for Action Development

File Path	Responsibility
[lib/types/action.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/types/action.ts)	Runtime re-export shim maintaining backward compatibility
packages/@openmaic/dsl/src/action.ts	Canonical type definitions with execution category constants (FIRE_AND_FORGET_ACTIONS, SYNC_ACTIONS, SLIDE_ONLY_ACTIONS)
[lib/playback/action-navigation.ts](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/playback/action-navigation.ts)	Playback engine consuming Action types during lesson execution
[components/workbench/chat/action-cluster.tsx](https://github.com/THU-MAIC/OpenMAIC/blob/main/components/workbench/chat/action-cluster.tsx)	UI component rendering action clusters in teacher chat panels

Execution Category Constants

The DSL package exports three helper constants for runtime classification:

• FIRE_AND_FORGET_ACTIONS — Array containing 'spotlight', 'laser'
• SYNC_ACTIONS — Array of all 15 blocking action types
• SLIDE_ONLY_ACTIONS — Subset restricted to slide-level contexts

These constants enable pattern-matching in lib/playback/action-navigation.ts without hardcoding type strings throughout the codebase.

Summary

• OpenMAIC lib/action supports 22 classroom interaction action types across three execution categories: fire-and-forget (2), synchronous (15), and widget interaction (4)
• All types are defined in @openmaic/dsl and re-exported through lib/types/action.ts to maintain a single source of truth
• Synchronous whiteboard actions (wb_*) form the largest category, enabling comprehensive annotation and drawing capabilities
• Widget interaction actions provide stateful manipulation of embedded interactive components
• Execution category constants (FIRE_AND_FORGET_ACTIONS, SYNC_ACTIONS) facilitate runtime behavior discrimination in the playback engine

Frequently Asked Questions

How do I determine if an action blocks lesson progression?

Check the SYNC_ACTIONS constant exported from @openmaic/dsl. Actions in this array—including all wb_*, speech, play_video, and discussion types—must complete before the playback engine advances. Fire-and-forget actions (spotlight, laser) execute in parallel.

Can I extend lib/action with custom action types?

The Action union in packages/@openmaic/dsl/src/action.ts is the extension point. Add your interface definition there, update the union type, and modify lib/playback/action-navigation.ts to handle execution. Re-export through lib/types/action.ts to maintain import consistency.

What distinguishes widget_setState from widget_highlight?

widget_setState modifies internal widget state (data model, configuration) and requires the widget to implement state handlers. widget_highlight applies transient visual effects (border glow, pulse animation) without changing widget internals—purely presentational feedback for classroom attention direction.

Where does the runtime enforce action type discrimination?

The playback engine at lib/playback/action-navigation.ts consumes Action unions and dispatches to specialized handlers based on action.type. TypeScript's exhaustiveness checking ensures all 22 action types have corresponding implementation branches, preventing unhandled action failures at runtime.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}