Image-to-Video (I2V) Generation Workflow in Open-Generative-AI Studio

The I2V workflow transforms a static image into animated video by uploading to the Video Studio, configuring model-specific parameters in VideoStudio.js, and calling muapi.generateI2V() to poll the MU-API until the video URL returns.

The Open-Generative-AI repository by Anil-matcha provides a complete Video Studio interface for converting images to video using various I2V models. This client-side workflow handles everything from upload to final video display, leveraging the remote MU-API for the actual synthesis while managing state and history locally. Understanding this generation pipeline is essential for developers integrating image-to-video capabilities into their own applications.

Step-by-Step I2V Generation Process

The Image-to-Video workflow follows a strict nine-step pipeline implemented entirely within the browser:

• Upload a reference image – The createUploadPicker component captures the image URL and stores it in uploadedImageUrl.
• Switch to I2V mode – When an image is selected, the internal flag imageMode is set to true and the first I2V model (i2vModels[0]) becomes the active selectedModel.
• Configure I2V parameters – Aspect ratio, duration, resolution, quality, mode, and effect name populate from the model’s inputs via helpers in src/lib/models.js. UI dropdowns (arBtn, durationBtn, resolutionBtn, qualityBtn, modeBtn, effectNameBtn) allow user customization.
• Trigger generation – Clicking Generate runs the async handler on generateBtn, which validates inputs and ensures an API key exists.
• Build the request payload – An object i2vParams is assembled containing model, image_url, prompt, aspect_ratio, duration, resolution, quality, mode, and name.
• Call the MU-API – muapi.generateI2V(i2vParams) sends the request. A request_id is saved via savePendingJob to enable polling.
• Poll for the result – muapi.pollForResult repeatedly checks the job status until a video URL is returned.
• Show the video – Once the URL arrives, showVideoInCanvas(url, selectedModel) hides the prompt UI, displays the video element, and enables the Extend button for Seedance 2.0 models.
• Persist history – The video entry is stored in localStorage under video_history and shown in the side-panel (historySidebar).

Core Files and Architecture

The I2V workflow spans several modular components in the src directory:

VideoStudio.js

Located at src/components/VideoStudio.js, this 1129-line component manages the full UI and workflow for text-to-video, image-to-video, video-to-video, and extend mode. It handles state management, dropdown rendering, generation logic, and history tracking. Key sections include lines 84-99 for upload handling and lines 220-256 for the generation and display logic.

models.js

The src/lib/models.js file contains an auto-generated catalogue of all models including I2V variants. It provides helper functions such as getCurrentAspectRatios, getCurrentDurations, getCurrentResolutions, getCurrentModes, and getEffectNamesForModel to populate the UI controls dynamically based on the selected model’s metadata.

muapi.js

Found in src/lib/muapi.js, this thin wrapper manages MU-API endpoints including generateI2V, generateVideo, processV2V, and pollForResult. It handles request-ID management and coordinates with the pending job storage system to resume generations after page reloads.

pendingJobs.js

The src/lib/pendingJobs.js module provides simple local storage utilities (savePendingJob, removePendingJob, getPendingJobs) that record pending jobs using the request_id returned by the API.

Constructing the I2V API Request

The frontend constructs the payload programmatically before sending it to muapi.generateI2V. Below is a minimal implementation based on the actual studio logic:

import { muapi } from '../lib/muapi.js';
import { i2vModels, getAspectRatiosForI2VModel,
         getDurationsForI2VModel, getResolutionsForI2VModel,
         getModesForModel, getEffectNamesForModel } from '../lib/models.js';

// 1. Select the default I2V model
const model = i2vModels[0];
const modelId = model.id;

// 2. Gather parameters from user input or defaults
const imageUrl = 'https://example.com/start-frame.png';
const prompt   = 'A sunrise over a futuristic city, cinematic motion';
const ar       = getAspectRatiosForI2VModel(modelId)[0];   // e.g. "16:9"
const dur      = getDurationsForI2VModel(modelId)[0];     // e.g. 5 seconds
const res      = getResolutionsForI2VModel(modelId)[0];   // e.g. "720p"
const quality  = model.inputs?.quality?.enum?.[0] ?? '';
const mode     = getModesForModel(modelId)[0] ?? '';
const effect   = getEffectNamesForModel(modelId)[0] ?? '';

// 3. Assemble the request payload
const i2vParams = {
  model: modelId,
  image_url: imageUrl,
  prompt,
  aspect_ratio: ar,
  duration: dur,
  resolution: res,
  quality,
  mode,
  name: effect
};

// 4. Capture request_id for pending-job tracking
let requestId = null;
const onRequestId = rid => { 
  requestId = rid; 
  // Store pending job for page-reload resilience
};

// 5. Execute the generation
muapi.generateI2V({ ...i2vParams, onRequestId })
  .then(res => {
    if (res?.url) {
      console.log('Video ready:', res.url);
      // Add to history and display in canvas
    }
  })
  .catch(err => console.error('I2V generation failed', err));

Key constraints: The image_url field is mandatory for all I2V requests, while prompt is optional—omitting it produces a purely motion-driven video. All other parameters must match the enumerations defined in the model’s inputs object.

Handling Asynchronous Results and Polling

Because video generation is asynchronous, the studio implements a robust polling mechanism. When muapi.generateI2V initiates a request, it immediately returns a request_id via the onRequestId callback. The system stores this ID using savePendingJob in localStorage, allowing the UI to resume polling if the user refreshes the page.

The muapi.pollForResult function repeatedly queries the job status until the response contains a url field. Once detected, removePendingJob clears the stored ID, and showVideoInCanvas swaps the UI sections to display the player. For Seedance 2.0 models specifically, the logic at lines 386-393 in VideoStudio.js enables the Extend button, allowing chained generations that use the output as input for subsequent clips.

Summary

• The I2V workflow is fully client-side, with heavy processing delegated to the remote MU-API.
• Required fields for generation include model, image_url, and model-specific parameters sourced from src/lib/models.js.
• State persistence uses localStorage for both pending jobs (request_id) and video history (video_history).
• Polling logic in muapi.js handles asynchronous completion, enabling seamless UX across page reloads.
• Seedance 2.0 models support the Extend feature for creating longer sequences from a single image.

Frequently Asked Questions

What parameters are mandatory for an I2V generation request?

The only mandatory parameter is image_url, which must point to the reference image you want to animate. While the prompt field can be an empty string (yielding motion-only generation), all other parameters—aspect_ratio, duration, resolution, quality, mode, and name—must be populated from the selected model’s inputs definition in src/lib/models.js to ensure compatibility with the specific I2V model endpoint.

How does the studio handle long-running video generations?

The system uses muapi.pollForResult to check the job status repeatedly until completion. To prevent data loss during page reloads, the request_id returned by the initial API call is immediately stored via savePendingJob in src/lib/pendingJobs.js. On page load, getPendingJobs retrieves any unfinished work, and the UI automatically resumes polling for the video URL without requiring the user to restart the generation.

Can I extend the generated video output?

Yes, but only when using Seedance 2.0 models. According to VideoStudio.js lines 386-393, the Extend button becomes visible in the UI specifically for these models. Clicking it initiates a new generation workflow that uses the previously generated video as the input reference, allowing creators to create longer sequences from a single starting image.

Where is the generation history stored?

Completed video metadata—including the URL, model ID, prompt, and parameters—is stored in the browser’s localStorage under the key video_history. The historySidebar component reads this storage to populate the side-panel interface, providing quick access to previous generations without requiring server-side session management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}