What Types of Interactive UI Elements Does OpenMAIC Support Beyond Standard Slides?

OpenMAIC supports three specialized interactive UI paradigms: dynamic interactive HTML scenes for embedding full widget pages, interactive element references that capture specific clickable DOM nodes inside iframes, and static interactive HTML for frozen offline-capable widgets.

The OpenMAIC (Machine-Agent Interactive Content) framework distinguishes between static slide objects and dynamic interactive layers that enable user manipulation. While standard slides contain immutable elements like text boxes and images, the repository's lib/workbench/element-refs.ts and video-export modules implement APIs for living UI components. This article examines the specific interactive element types available in the THU-MAIC/OpenMAIC codebase, including their implementation details and usage patterns.

Interactive HTML Scenes

Interactive HTML scenes represent self-contained web pages or widgets that render inside the presentation timeline. Unlike static slide objects, these scenes accept HTML content and execute JavaScript within an isolated iframe environment.

In the OpenMAIC source, scenes are defined with the discriminator type: 'interactive', as demonstrated in the video-export test suites. These scenes provide the container for complex UI elements such as buttons, forms, and mini-applications that require DOM interaction.

Defining an Interactive Scene

The following pattern creates an interactive scene for video compilation:

const interactiveScene = {
  id: 'interactive:widget',
  type: 'interactive',
  content: { type: 'interactive', html: '<!doctype html><h1>Widget</h1>' },
};

await compileVideoTimeline({
  scenes: [interactiveScene],
  // ...other timeline options
});

The compiled output places the widget at assets/interactive/001-widget.html, which the playback engine loads as a self-contained interactive unit.

Interactive Element References

While interactive scenes host full HTML pages, interactive element references capture specific, clickable DOM nodes within those iframes. The InteractiveElementRef type, defined in lib/workbench/element-refs.ts, records a node's CSS selector, outer HTML, visible text, and localized label.

When a user interacts with the referenced element, the system emits a "click" event that the agent can convert into messages or edit commands. This mechanism allows the LLM to interact with granular UI components rather than entire pages.

Creating Interactive Element References

Use the makeInteractiveElementRef factory function to generate references from user selections:

import { makeInteractiveElementRef } from '@/lib/workbench/element-refs';

// Capture a button selected inside an interactive iframe
const picked = {
  selector: '#cta-button',
  outerHTML: '<button id="cta-button">Click me</button>',
  text: 'Click me',
};

const interactiveRef = makeInteractiveElementRef(
  'stage-01',          // stageId
  'scene-interactive-01', // sceneId
  picked,
  (key) => key,        // translation function placeholder
);

// Resulting object structure:
// {
//   kind: 'interactive-element',
//   stageId: 'stage-01',
//   sceneId: 'scene-interactive-01',
//   selector: '#cta-button',
//   outerHTML: '<button id="cta-button">Click me</button>',
//   text: 'Click me',
//   label: 'button · Click me',
// }

The resulting reference includes kind: 'interactive-element', distinguishing it from standard slide object references according to the implementation in tests/workbench/element-refs.test.ts.

Static Interactive HTML

For offline playback and archival purposes, OpenMAIC supports static interactive HTML—frozen snapshots of interactive widgets. The prepareInteractiveHtmlScenes utility processes dynamic widgets into self-contained ZIP packages that retain their visual state without active server connections.

These static variants include diagnostic attributes like data-interactive-static-state to indicate whether the widget is frozen, active, or has fallen back to a degraded state. This functionality is validated in tests/video-export/interactive-static-html.browser.test.ts, which ensures static widgets render correctly in Chromium environments.

Standard Slides vs. Interactive UI

Understanding the distinction between static slides and interactive elements is crucial for content architecture:

• Standard slide elements include text, images, shapes, and charts that render as static objects without DOM accessibility or event handling.
• Interactive HTML scenes provide full iframe encapsulation for custom web applications, supporting JavaScript execution and CSS animations through the type: 'interactive' discriminator.
• Interactive element references enable granular agent interaction by targeting specific selectors within those iframes, bridging the gap between visual presentation and functional UI manipulation.
• Static interactive HTML packages these dynamic experiences into portable, offline-ready formats suitable for video export and distribution.

Summary

• Interactive HTML scenes embed full widget pages via the type: 'interactive' property, providing isolated execution environments for custom UI in lib/workbench/element-refs.ts.
• Interactive element references (InteractiveElementRef) capture specific clickable DOM nodes using makeInteractiveElementRef, enabling precise agent interaction through CSS selectors.
• Static interactive HTML freezes widgets for offline use via prepareInteractiveHtmlScenes, marked with data-interactive-static-state attributes for runtime state management.
• These implementations reside in the THU-MAIC/OpenMAIC video-export test suite and element-refs modules, distinguishing OpenMAIC's capabilities from traditional static slide objects.

Frequently Asked Questions

What distinguishes an interactive element reference from a standard slide object?

Interactive element references target specific DOM nodes within iframes, storing CSS selectors and outer HTML in the InteractiveElementRef structure, whereas standard slide objects represent static, non-interactive content like text boxes or images without underlying DOM accessibility.

How does OpenMAIC capture user interactions on dynamic UI elements?

When a user clicks an element captured by makeInteractiveElementRef, the system extracts the selector and text content from the iframe, creating a reference that emits events the LLM can process as edit commands or messages, as implemented in lib/workbench/element-refs.ts.

What is the purpose of static interactive HTML in OpenMAIC?

Static interactive HTML provides frozen, self-contained versions of widgets via prepareInteractiveHtmlScenes, allowing presentations to function offline or within exported video timelines while maintaining diagnostic state markers like data-interactive-static-state.

Can interactive HTML scenes execute arbitrary JavaScript?

Yes, interactive scenes defined with type: 'interactive' encapsulate full HTML documents within iframes, enabling execution of JavaScript, CSS animations, and complex form handling, validated through the browser-specific tests in tests/video-export/interactive-static-html.browser.test.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}