How the `play_video` Action Works in OpenMAIC: A Technical Deep Dive

OpenMAIC's playvideo action enables generative AI to trigger video playback by dispatching JSON commands that update a Zustand store, which reactive UI components consume to render native HTML5 video elements.

The play_video action in OpenMAIC bridges large language model outputs with frontend media presentation. This lightweight command structure allows the AI to programmatically control video playback within the workspace interface by manipulating a centralized state container. Understanding this mechanism reveals how OpenMAIC transforms LLM decisions into interactive multimedia experiences.

Action Architecture Overview

OpenMAIC implements a registry-based action system that maps string identifiers to handler functions. When the LLM emits a command, the dispatcher in app/workspace/actionRegistry.ts resolves the action name and executes the corresponding handler with parsed arguments. This decouples AI logic from UI implementation, allowing new capabilities to be added without modifying the core conversation flow.

The playvideo Action Implementation

Action Registration in actionRegistry.ts

The action registry declares available capabilities to the LLM and binds them to executable code. In app/workspace/actionRegistry.ts, the playvideo action is registered alongside other workspace tools:

// app/workspace/actionRegistry.ts
export const actionRegistry = {
  playvideo: {
    handler: playvideo,
    schema: z.object({
      url: z.string().url(),
      start: z.number().optional()
    }),
    description: "Play a video from the provided URL"
  }
};

This registration exposes the action to the AI system and validates incoming arguments using Zod schemas before execution.

Handler Logic in playvideo.ts

The handler file app/workspace/actions/playvideo.ts contains the core execution logic. It extracts the video URL and optional start timestamp, then commits these values to the global video store:

// app/workspace/actions/playvideo.ts
import { useVideoStore } from '@/stores/videoStore';
import { addNotice } from '@/stores/noticeStore';

export async function playvideo(args: { url: string; start?: number }) {
  const { url, start = 0 } = args;
  try {
    const setVideo = useVideoStore.getState().setVideo;
    setVideo({ url, start });
  } catch (e) {
    addNotice({ type: 'error', message: `Unable to play video: ${e}` });
  }
}

The handler performs no direct DOM manipulation; it acts purely as a state mutator, delegating rendering concerns to subscribed React components.

State Management via videoStore.ts

OpenMAIC uses Zustand for client-side state management. The stores/videoStore.ts file defines the video state slice that persists the current playback parameters:

// stores/videoStore.ts
import { create } from 'zustand';

interface VideoState {
  current: { url: string; start: number } | null;
  setVideo: (params: { url: string; start: number }) => void;
  clearVideo: () => void;
}

export const useVideoStore = create<VideoState>((set) => ({
  current: null,
  setVideo: (params) => set({ current: params }),
  clearVideo: () => set({ current: null })
}));

When setVideo is invoked, React's reconciliation process triggers re-renders in all components subscribed to this store.

UI Rendering in VideoPlayer.tsx

The components/VideoPlayer.tsx component subscribes to useVideoStore and manages the actual HTML5 video element. It uses a useEffect hook to respond to state changes by loading the source, setting the current time, and initiating playback:

// components/VideoPlayer.tsx
import { useEffect, useRef } from 'react';
import { useVideoStore } from '@/stores/videoStore';

export const VideoPlayer = () => {
  const { url, start } = useVideoStore(state => state.current) || {};
  const videoRef = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    if (videoRef.current && url) {
      videoRef.current.src = url;
      videoRef.current.currentTime = start || 0;
      videoRef.current.play().catch(err => {
        console.error('Playback failed:', err);
      });
    }
  }, [url, start]);

  if (!url) return null;

  return (
    <div className="video-container">
      <video 
        ref={videoRef} 
        controls 
        style={{ width: '100%', maxHeight: '600px' }} 
      />
    </div>
  );
};

This component handles the imperative DOM API calls that are impossible to perform directly from the action handler, maintaining proper separation between state logic and media controls.

JSON Payload Structure and Usage

When the LLM decides to trigger video playback, it emits a structured JSON object that conforms to the Zod schema defined in the registry:

{
  "action": "playvideo",
  "args": {
    "url": "https://cdn.example.com/tutorials/intro.mp4",
    "start": 12
  }
}

The url parameter must be a valid HTTP or HTTPS address accessible to the client browser. The optional start parameter accepts a number representing seconds from the beginning of the video. If omitted, playback begins at the zero-second mark.

Error Handling Mechanisms

If the video fails to load or the URL is unreachable, the handler catches exceptions and dispatches error notifications through stores/noticeStore.ts. The addNotice function pushes messages to a toast queue that renders ephemeral UI alerts:

// stores/noticeStore.ts
interface Notice {
  type: 'error' | 'info' | 'success';
  message: string;
  id: string;
}

export const useNoticeStore = create((set) => ({
  notices: [],
  addNotice: (notice: Omit<Notice, 'id'>) => 
    set((state) => ({ 
      notices: [...state.notices, { ...notice, id: crypto.randomUUID() }] 
    }))
}));

This ensures users receive immediate visual feedback when AI-triggered media content fails to render, preventing silent failures in the workspace.

Summary

• OpenMAIC actions follow a registry pattern where string identifiers map to handler functions in app/workspace/actionRegistry.ts.
• The playvideo handler updates Zustand state rather than manipulating the DOM directly, maintaining React best practices.
• State synchronization occurs through stores/videoStore.ts, which broadcasts changes to subscribed components like components/VideoPlayer.tsx.
• Playback initiation happens imperatively via HTMLMediaElement.play() inside a useEffect hook that responds to store updates.
• Error resilience is implemented through stores/noticeStore.ts, which displays toast notifications for failed video loads.

Frequently Asked Questions

What parameters does the playvideo action accept?

The action accepts a JSON object with url (required string) and start (optional number) properties. The URL must resolve to a valid video file accessible via the client's network, while the start time specifies the initial playback position in seconds.

How does OpenMAIC handle video playback errors?

Errors are caught in the playvideo handler within app/workspace/actions/playvideo.ts and propagated to the user via addNotice from stores/noticeStore.ts. This displays a toast notification without crashing the workspace application.

Can the playvideo action stream videos from external URLs?

Yes, the action supports any URL accessible to the browser, including external CDNs and signed URLs. The component uses standard HTML5 video element loading mechanisms, respecting CORS policies and authentication headers present in the browser session.

Is the video playback synchronized across multiple users?

No, the playvideo action updates only the local client's useVideoStore instance. OpenMAIC does not implement real-time state synchronization for media playback across collaborative sessions; each user receives independent video instances when the AI triggers the action.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}