How to Extend Widget Payloads and Action Sets Using Scene's Generic Parameters in OpenMAIC

Applications can extend OpenMAIC's widget payloads and action sets by supplying custom generic arguments to the Scene type defined in packages/@openmaic/dsl/src/stage.ts, enabling type-safe augmentation of both content contracts and playback actions without modifying core library code.

The OpenMAIC framework provides a flexible lesson model centered around the generic Scene type. By leveraging its generic parameters TAction and TContent, developers can inject custom widget payloads and specialized playback actions while maintaining strict TypeScript type safety. This approach allows applications to enrich interactive scenes with proprietary content types and custom behaviors without forking the underlying DSL package.

Understanding the Scene Generic Type

The foundation of extensibility lies in the Scene type definition located at lines 78‑84 in /packages/@openmaic/dsl/src/stage.ts. The type signature uses two generic parameters with sensible defaults:

export type Scene<
  TAction = Action,
  TContent extends { type: SceneType } = SlideContent | QuizContent,
> = TContent extends unknown
  ? SceneCore<TAction> & { type: TContent['type']; content: TContent }
  : never;

The first parameter TAction represents the union of playback actions available within the scene. By default, it resolves to the standard Action union containing built‑in actions such as speech and widget_highlight.

The second parameter TContent constrains the shape of the scene's payload. It defaults to SlideContent | QuizContent, but accepts any discriminated union where each member carries a type: SceneType property. This design enables the compiler to statically enforce relationships between scene.type and scene.content.

As noted in the source comments on lines 10‑11, these generic parameters explicitly support application‑level extension of both widget payloads and action sets.

Extending Widget Payloads with TContent

To introduce a custom widget type, define a new content interface that satisfies the SceneType discriminant constraint. This pattern allows the renderer, import/export pipeline, and custom plugins to recognize and process richer payload structures.

Define your custom payload by extending the content contract with widget‑specific fields:

// Define a custom widget content type
interface DiagramWidgetContent {
  type: 'interactive';                 // Must satisfy SceneType discriminant
  widgetType: 'diagram';
  widgetConfig: { 
    nodes: { id: string; label: string }[] 
  };
}

This interface adds widgetType and widgetConfig fields to the standard scene content. When passed as TContent, TypeScript narrows the scene.content type based on the scene.type discriminator, ensuring that code accessing widgetConfig only compiles when the type is 'interactive'.

Extending Action Sets with TAction

Custom playback behaviors require extending the action union. By creating a new union type that includes both the default Action members and your custom action shapes, you can add specialized commands like state management or widget‑specific triggers.

Extend the action set by unioning custom actions with the base Action type:

// Extend the standard action union with widget-specific actions
type ExtendedAction =
  | Action                               // Keep existing actions
  | { type: 'widget_setState'; state: Record<string, unknown> }
  | { type: 'widget_reveal'; targetId: string };

This ExtendedAction union preserves all standard OpenMAIC actions while adding widget_setState and widget_reveal. When used as TAction, this guarantees that the actions array in any scene instance only contains valid, type‑checked commands.

Implementing Custom Scenes in Practice

Combine both extensions by instantiating Scene with your custom generics. This creates a concrete type that governs both the content payload and the permissible actions throughout your application.

Create the specialized scene type and instantiate it:

// Create a concrete Scene type using both extensions
type AppScene = Scene<
  ExtendedAction, 
  SlideContent | QuizContent | DiagramWidgetContent
>;

// Example instance containing a diagram widget and custom actions
const diagramScene: AppScene = {
  id: 'scene-01',
  stageId: 'stage-01',
  title: 'Introduction to Graphs',
  order: 0,
  type: 'interactive',
  content: {
    type: 'interactive',
    widgetType: 'diagram',
    widgetConfig: {
      nodes: [
        { id: 'n1', label: 'Node 1' }, 
        { id: 'n2', label: 'Node 2' }
      ],
    },
  },
  actions: [
    { type: 'speech', text: 'Let’s explore this diagram together.' },
    { type: 'widget_setState', state: { highlightedNode: 'n1' } },
  ],
};

The AppScene type statically enforces that when type is 'interactive', the content field must contain DiagramWidgetContent properties, and that every element in actions conforms to ExtendedAction. This prevents runtime errors when the workbench or renderer processes the scene.

Integration with OpenMAIC Tooling

Several files in the OpenMAIC repository demonstrate how extended scenes integrate with the broader ecosystem:

• lib/workbench/composer-keys.ts illustrates how the UI composes action sets for scenes, providing a template for wiring custom actions into the workbench interface.
• tests/workbench/tts-stage-sync.test.ts contains Scene builders used in integration tests, showing practical examples of generic‑typed scene construction in test environments.

These references demonstrate that the generic Scene type is consumed not only by the DSL package but also by the workbench and testing utilities, ensuring consistency across the entire application stack.

Summary

• The Scene type in packages/@openmaic/dsl/src/stage.ts accepts two generic parameters: TAction for playback actions and TContent for content payloads.
• Extending widget payloads involves defining new content interfaces with type: SceneType discriminants and passing them as TContent.
• Extending action sets requires building a union type that adds custom actions to the base Action union and supplying it as TAction.
• TypeScript statically enforces relationships between scene types, content shapes, and action arrays, eliminating runtime type errors.
• The pattern requires no modifications to the core OpenMAIC library, enabling safe upgrades and modular plugin architectures.

Frequently Asked Questions

What is the default Scene configuration in OpenMAIC?

By default, Scene resolves to Scene<Action, SlideContent | QuizContent>, where Action represents the standard playback actions (speech, highlights, etc.) and the content union covers basic slide and quiz content types. These defaults provide immediate usability while remaining fully extensible through generic parameters.

Do I need to fork the OpenMAIC DSL package to add custom widgets?

No. Because Scene is implemented as a generic type with unconstrained type parameters, applications can define custom TContent and TAction arguments externally. The compiler validates these extensions at the application level without requiring changes to packages/@openmaic/dsl/src/stage.ts.

How does TypeScript enforce type safety with extended Scenes?

TypeScript uses the discriminant type field in TContent to narrow the content property based on the scene.type value. Simultaneously, it validates the actions array against TAction. This means accessing scene.content.widgetConfig only compiles when scene.type matches the custom content type, and pushing invalid actions to the array triggers compile‑time errors.

Where can I find implementation examples of extended Scenes in the OpenMAIC codebase?

The test suite in tests/workbench/tts-stage-sync.test.ts demonstrates generic scene construction patterns, while lib/workbench/composer-keys.ts shows how the workbench handles action composition. These files provide concrete templates for implementing custom widget payloads and action unions in production applications.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}