# How the `play_video` Action Works in OpenMAIC: A Technical Deep Dive

> Explore OpenMAIC's play_video action. Learn how generative AI triggers video playback via JSON commands, updating a Zustand store for reactive UI rendering of HTML5 video.

- Repository: [MAIC/OpenMAIC](https://github.com/THU-MAIC/OpenMAIC)
- Tags: deep-dive
- Published: 2026-09-05

---

**OpenMAIC's `playvideo` action enables generative AI to trigger video playback by dispatching JSON commands that update a Zustand store, which reactive UI components consume to render native HTML5 video elements.**

The `play_video` action in OpenMAIC bridges large language model outputs with frontend media presentation. This lightweight command structure allows the AI to programmatically control video playback within the workspace interface by manipulating a centralized state container. Understanding this mechanism reveals how OpenMAIC transforms LLM decisions into interactive multimedia experiences.

## Action Architecture Overview

OpenMAIC implements a registry-based action system that maps string identifiers to handler functions. When the LLM emits a command, the dispatcher in [`app/workspace/actionRegistry.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/app/workspace/actionRegistry.ts) resolves the action name and executes the corresponding handler with parsed arguments. This decouples AI logic from UI implementation, allowing new capabilities to be added without modifying the core conversation flow.

## The playvideo Action Implementation

### Action Registration in actionRegistry.ts

The action registry declares available capabilities to the LLM and binds them to executable code. In [`app/workspace/actionRegistry.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/app/workspace/actionRegistry.ts), the `playvideo` action is registered alongside other workspace tools:

```typescript
// app/workspace/actionRegistry.ts
export const actionRegistry = {
  playvideo: {
    handler: playvideo,
    schema: z.object({
      url: z.string().url(),
      start: z.number().optional()
    }),
    description: "Play a video from the provided URL"
  }
};

```

This registration exposes the action to the AI system and validates incoming arguments using Zod schemas before execution.

### Handler Logic in playvideo.ts

The handler file [`app/workspace/actions/playvideo.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/app/workspace/actions/playvideo.ts) contains the core execution logic. It extracts the video URL and optional start timestamp, then commits these values to the global video store:

```typescript
// app/workspace/actions/playvideo.ts
import { useVideoStore } from '@/stores/videoStore';
import { addNotice } from '@/stores/noticeStore';

export async function playvideo(args: { url: string; start?: number }) {
  const { url, start = 0 } = args;
  try {
    const setVideo = useVideoStore.getState().setVideo;
    setVideo({ url, start });
  } catch (e) {
    addNotice({ type: 'error', message: `Unable to play video: ${e}` });
  }
}

```

The handler performs no direct DOM manipulation; it acts purely as a state mutator, delegating rendering concerns to subscribed React components.

### State Management via videoStore.ts

OpenMAIC uses Zustand for client-side state management. The [`stores/videoStore.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stores/videoStore.ts) file defines the video state slice that persists the current playback parameters:

```typescript
// stores/videoStore.ts
import { create } from 'zustand';

interface VideoState {
  current: { url: string; start: number } | null;
  setVideo: (params: { url: string; start: number }) => void;
  clearVideo: () => void;
}

export const useVideoStore = create<VideoState>((set) => ({
  current: null,
  setVideo: (params) => set({ current: params }),
  clearVideo: () => set({ current: null })
}));

```

When `setVideo` is invoked, React's reconciliation process triggers re-renders in all components subscribed to this store.

### UI Rendering in VideoPlayer.tsx

The [`components/VideoPlayer.tsx`](https://github.com/THU-MAIC/OpenMAIC/blob/main/components/VideoPlayer.tsx) component subscribes to `useVideoStore` and manages the actual HTML5 video element. It uses a `useEffect` hook to respond to state changes by loading the source, setting the current time, and initiating playback:

```tsx
// components/VideoPlayer.tsx
import { useEffect, useRef } from 'react';
import { useVideoStore } from '@/stores/videoStore';

export const VideoPlayer = () => {
  const { url, start } = useVideoStore(state => state.current) || {};
  const videoRef = useRef<HTMLVideoElement>(null);

  useEffect(() => {
    if (videoRef.current && url) {
      videoRef.current.src = url;
      videoRef.current.currentTime = start || 0;
      videoRef.current.play().catch(err => {
        console.error('Playback failed:', err);
      });
    }
  }, [url, start]);

  if (!url) return null;

  return (
    <div className="video-container">
      <video 
        ref={videoRef} 
        controls 
        style={{ width: '100%', maxHeight: '600px' }} 
      />
    </div>
  );
};

```

This component handles the imperative DOM API calls that are impossible to perform directly from the action handler, maintaining proper separation between state logic and media controls.

## JSON Payload Structure and Usage

When the LLM decides to trigger video playback, it emits a structured JSON object that conforms to the Zod schema defined in the registry:

```json
{
  "action": "playvideo",
  "args": {
    "url": "https://cdn.example.com/tutorials/intro.mp4",
    "start": 12
  }
}

```

The `url` parameter must be a valid HTTP or HTTPS address accessible to the client browser. The optional `start` parameter accepts a number representing seconds from the beginning of the video. If omitted, playback begins at the zero-second mark.

## Error Handling Mechanisms

If the video fails to load or the URL is unreachable, the handler catches exceptions and dispatches error notifications through [`stores/noticeStore.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stores/noticeStore.ts). The `addNotice` function pushes messages to a toast queue that renders ephemeral UI alerts:

```typescript
// stores/noticeStore.ts
interface Notice {
  type: 'error' | 'info' | 'success';
  message: string;
  id: string;
}

export const useNoticeStore = create((set) => ({
  notices: [],
  addNotice: (notice: Omit<Notice, 'id'>) => 
    set((state) => ({ 
      notices: [...state.notices, { ...notice, id: crypto.randomUUID() }] 
    }))
}));

```

This ensures users receive immediate visual feedback when AI-triggered media content fails to render, preventing silent failures in the workspace.

## Summary

- **OpenMAIC actions** follow a registry pattern where string identifiers map to handler functions in [`app/workspace/actionRegistry.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/app/workspace/actionRegistry.ts).
- The **`playvideo` handler** updates Zustand state rather than manipulating the DOM directly, maintaining React best practices.
- **State synchronization** occurs through [`stores/videoStore.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stores/videoStore.ts), which broadcasts changes to subscribed components like [`components/VideoPlayer.tsx`](https://github.com/THU-MAIC/OpenMAIC/blob/main/components/VideoPlayer.tsx).
- **Playback initiation** happens imperatively via `HTMLMediaElement.play()` inside a `useEffect` hook that responds to store updates.
- **Error resilience** is implemented through [`stores/noticeStore.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stores/noticeStore.ts), which displays toast notifications for failed video loads.

## Frequently Asked Questions

### What parameters does the playvideo action accept?

The action accepts a JSON object with `url` (required string) and `start` (optional number) properties. The URL must resolve to a valid video file accessible via the client's network, while the start time specifies the initial playback position in seconds.

### How does OpenMAIC handle video playback errors?

Errors are caught in the `playvideo` handler within [`app/workspace/actions/playvideo.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/app/workspace/actions/playvideo.ts) and propagated to the user via `addNotice` from [`stores/noticeStore.ts`](https://github.com/THU-MAIC/OpenMAIC/blob/main/stores/noticeStore.ts). This displays a toast notification without crashing the workspace application.

### Can the playvideo action stream videos from external URLs?

Yes, the action supports any URL accessible to the browser, including external CDNs and signed URLs. The component uses standard HTML5 video element loading mechanisms, respecting CORS policies and authentication headers present in the browser session.

### Is the video playback synchronized across multiple users?

No, the `playvideo` action updates only the local client's `useVideoStore` instance. OpenMAIC does not implement real-time state synchronization for media playback across collaborative sessions; each user receives independent video instances when the AI triggers the action.