# How to Extend Widget Payloads and Action Sets Using Scene's Generic Parameters in OpenMAIC

> Learn how to extend OpenMAIC widget payloads and action sets using Scene's generic parameters. Augment content contracts and playback actions type-safely without modifying core code.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: how-to-guide
- Published: 2026-09-04

---

**Applications can extend OpenMAIC's widget payloads and action sets by supplying custom generic arguments to the `Scene` type defined in `packages/@openmaic/dsl/src/stage.ts`, enabling type-safe augmentation of both content contracts and playback actions without modifying core library code.**

The OpenMAIC framework provides a flexible lesson model centered around the generic `Scene` type. By leveraging its generic parameters `TAction` and `TContent`, developers can inject custom widget payloads and specialized playback actions while maintaining strict TypeScript type safety. This approach allows applications to enrich interactive scenes with proprietary content types and custom behaviors without forking the underlying DSL package.

## Understanding the Scene Generic Type

The foundation of extensibility lies in the `Scene` type definition located at lines 78‑84 in `/packages/@openmaic/dsl/src/stage.ts`. The type signature uses two generic parameters with sensible defaults:

```typescript
export type Scene<
  TAction = Action,
  TContent extends { type: SceneType } = SlideContent | QuizContent,
> = TContent extends unknown
  ? SceneCore<TAction> & { type: TContent['type']; content: TContent }
  : never;

```

The **first parameter `TAction`** represents the union of playback actions available within the scene. By default, it resolves to the standard `Action` union containing built‑in actions such as `speech` and `widget_highlight`.

The **second parameter `TContent`** constrains the shape of the scene's payload. It defaults to `SlideContent | QuizContent`, but accepts any discriminated union where each member carries a `type: SceneType` property. This design enables the compiler to statically enforce relationships between `scene.type` and `scene.content`.

As noted in the source comments on lines 10‑11, these generic parameters explicitly support application‑level extension of both widget payloads and action sets.

## Extending Widget Payloads with TContent

To introduce a custom widget type, define a new content interface that satisfies the `SceneType` discriminant constraint. This pattern allows the renderer, import/export pipeline, and custom plugins to recognize and process richer payload structures.

Define your custom payload by extending the content contract with widget‑specific fields:

```typescript
// Define a custom widget content type
interface DiagramWidgetContent {
  type: 'interactive';                 // Must satisfy SceneType discriminant
  widgetType: 'diagram';
  widgetConfig: { 
    nodes: { id: string; label: string }[] 
  };
}

```

This interface adds `widgetType` and `widgetConfig` fields to the standard scene content. When passed as `TContent`, TypeScript narrows the `scene.content` type based on the `scene.type` discriminator, ensuring that code accessing `widgetConfig` only compiles when the type is `'interactive'`.

## Extending Action Sets with TAction

Custom playback behaviors require extending the action union. By creating a new union type that includes both the default `Action` members and your custom action shapes, you can add specialized commands like state management or widget‑specific triggers.

Extend the action set by unioning custom actions with the base `Action` type:

```typescript
// Extend the standard action union with widget-specific actions
type ExtendedAction =
  | Action                               // Keep existing actions
  | { type: 'widget_setState'; state: Record<string, unknown> }
  | { type: 'widget_reveal'; targetId: string };

```

This `ExtendedAction` union preserves all standard OpenMAIC actions while adding `widget_setState` and `widget_reveal`. When used as `TAction`, this guarantees that the `actions` array in any scene instance only contains valid, type‑checked commands.

## Implementing Custom Scenes in Practice

Combine both extensions by instantiating `Scene` with your custom generics. This creates a concrete type that governs both the content payload and the permissible actions throughout your application.

Create the specialized scene type and instantiate it:

```typescript
// Create a concrete Scene type using both extensions
type AppScene = Scene<
  ExtendedAction, 
  SlideContent | QuizContent | DiagramWidgetContent
>;

// Example instance containing a diagram widget and custom actions
const diagramScene: AppScene = {
  id: 'scene-01',
  stageId: 'stage-01',
  title: 'Introduction to Graphs',
  order: 0,
  type: 'interactive',
  content: {
    type: 'interactive',
    widgetType: 'diagram',
    widgetConfig: {
      nodes: [
        { id: 'n1', label: 'Node 1' }, 
        { id: 'n2', label: 'Node 2' }
      ],
    },
  },
  actions: [
    { type: 'speech', text: 'Let’s explore this diagram together.' },
    { type: 'widget_setState', state: { highlightedNode: 'n1' } },
  ],
};

```

The `AppScene` type statically enforces that when `type` is `'interactive'`, the `content` field must contain `DiagramWidgetContent` properties, and that every element in `actions` conforms to `ExtendedAction`. This prevents runtime errors when the workbench or renderer processes the scene.

## Integration with OpenMAIC Tooling

Several files in the OpenMAIC repository demonstrate how extended scenes integrate with the broader ecosystem:

- **[`lib/workbench/composer-keys.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/composer-keys.ts)** illustrates how the UI composes action sets for scenes, providing a template for wiring custom actions into the workbench interface.
- **[`tests/workbench/tts-stage-sync.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/tts-stage-sync.test.ts)** contains `Scene` builders used in integration tests, showing practical examples of generic‑typed scene construction in test environments.

These references demonstrate that the generic `Scene` type is consumed not only by the DSL package but also by the workbench and testing utilities, ensuring consistency across the entire application stack.

## Summary

- The `Scene` type in `packages/@openmaic/dsl/src/stage.ts` accepts two generic parameters: `TAction` for playback actions and `TContent` for content payloads.
- **Extending widget payloads** involves defining new content interfaces with `type: SceneType` discriminants and passing them as `TContent`.
- **Extending action sets** requires building a union type that adds custom actions to the base `Action` union and supplying it as `TAction`.
- TypeScript statically enforces relationships between scene types, content shapes, and action arrays, eliminating runtime type errors.
- The pattern requires no modifications to the core OpenMAIC library, enabling safe upgrades and modular plugin architectures.

## Frequently Asked Questions

### What is the default Scene configuration in OpenMAIC?

By default, `Scene` resolves to `Scene<Action, SlideContent | QuizContent>`, where `Action` represents the standard playback actions (speech, highlights, etc.) and the content union covers basic slide and quiz content types. These defaults provide immediate usability while remaining fully extensible through generic parameters.

### Do I need to fork the OpenMAIC DSL package to add custom widgets?

No. Because `Scene` is implemented as a generic type with unconstrained type parameters, applications can define custom `TContent` and `TAction` arguments externally. The compiler validates these extensions at the application level without requiring changes to `packages/@openmaic/dsl/src/stage.ts`.

### How does TypeScript enforce type safety with extended Scenes?

TypeScript uses the discriminant `type` field in `TContent` to narrow the `content` property based on the `scene.type` value. Simultaneously, it validates the `actions` array against `TAction`. This means accessing `scene.content.widgetConfig` only compiles when `scene.type` matches the custom content type, and pushing invalid actions to the array triggers compile‑time errors.

### Where can I find implementation examples of extended Scenes in the OpenMAIC codebase?

The test suite in [`tests/workbench/tts-stage-sync.test.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/tests/workbench/tts-stage-sync.test.ts) demonstrates generic scene construction patterns, while [`lib/workbench/composer-keys.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/lib/workbench/composer-keys.ts) shows how the workbench handles action composition. These files provide concrete templates for implementing custom widget payloads and action unions in production applications.