How Video/Screen Recording Mode Utilizes AI for Prototype Generation in Screenshot-to-Code

The video/screen recording mode captures a WebM clip via the browser's MediaRecorder API, converts it to a Base64 data-URL, and streams it through a WebSocket pipeline where Gemini 1.5 models analyze the visual frames and user interactions to synthesize a functional HTML/JS prototype.

The abi/screenshot-to-code repository enables developers to convert visual inputs into working frontend code. When utilizing the video/screen recording mode for AI prototype generation, the application leverages Google’s Gemini 1.5 models to process dynamic UI interactions frame-by-frame, creating interactive prototypes from screen recordings without any server-side video processing.

End-to-End AI Pipeline for Video Input

The implementation follows a strict pipeline from browser capture to AI-generated code, with each stage handling specific transformations of the video data.

Screen Capture and Data-URL Conversion

When a user initiates a recording, the ScreenRecorder component in frontend/src/components/recording/ScreenRecorder.tsx instantiates a MediaRecorder to collect video blobs. After the user stops recording, the component fixes the WebM duration and converts the final Blob to a Base64 data-URL using blobToBase64DataUrl.

// ScreenRecorder.tsx – after the recording stops
mediaRecorder.onstop = async () => {
  const completeBlob = await fixWebmDuration(
    new Blob(chunks, { type: "video/webm" })
  );
  const dataUrl = await blobToBase64DataUrl(completeBlob);
  setScreenRecordingDataUrl(dataUrl);
  setScreenRecorderState(ScreenRecorderState.FINISHED);
};

// Kick‑off generation
const kickoffGeneration = () => {
  if (screenRecordingDataUrl) {
    generateCode([screenRecordingDataUrl], "video");
  } else {
    toast.error("Screen recording does not exist.");
  }
};

WebSocket Transmission with Input Mode

The generateCode function in frontend/src/generateCode.ts opens a WebSocket connection to WS_BACKEND_URL/generate-code and transmits the video data-URL with inputMode explicitly set to "video". This flag triggers the backend’s video-specific processing logic.

export function generateCode(
  wsRef: React.MutableRefObject<WebSocket | null>,
  params: FullGenerationSettings,
  callbacks: CodeGenerationCallbacks
) {
  const ws = new WebSocket(`${WS_BACKEND_URL}/generate-code`);
  wsRef.current = ws;

  ws.addEventListener("open", () => ws.send(JSON.stringify(params)));
  // … handle streaming messages …
}

The params object includes:

{
  "inputMode": "video",
  "prompt": { "videos": ["data:video/webm;base64,…"] },
  "stack": "react"
}

Gemini-Only Model Selection

Upon receiving the WebSocket request, the backend forces the use of Gemini models exclusively. In backend/routes/generate_code.py, the ModelSelectionStage._get_variant_models method detects input_mode == "video" and returns only VIDEO_VARIANT_MODELS, which are Gemini 1.5 configurations. If no GEMINI_API_KEY is present, the request fails immediately.

# backend/routes/generate_code.py – Model selection

if input_mode == "video":
    if not gemini_api_key:
        raise Exception("Video mode requires a Gemini API key.")
    return list(VIDEO_VARIANT_MODELS)   # two Gemini variants

This design enables a side-by-side comparison of two different Gemini configurations (minimal and high-quality) for the same video input.

Video-Aware Prompt Construction

The prompt builder in backend/prompts/create/__init__.py routes video requests to build_video_prompt_messages in backend/prompts/create/video.py. This function constructs a ChatCompletion message array where the video data-URL is embedded as an image_url content part—a format that Gemini treats as a video URL. The system prompt instructs the model to watch the entire video, replicate the UI, and make it functional.

def build_video_prompt_messages(
    video_data_url: str,
    stack: Stack,
    text_prompt: str,
    image_generation_enabled: bool,
) -> list[ChatCompletionMessageParam]:
    user_content = [
        {"type": "image_url", "image_url": {"url": video_data_url, "detail": "high"}},
        {"type": "text", "text": f"""
You have been given a video of a user interacting with a web app...
{selected_stack}
""" + ("\nAdditional instructions: " + text_prompt if text_prompt.strip() else ""))}
    ]
    return [
        {"role": "system", "content": system_prompt.SYSTEM_PROMPT},
        {"role": "user",   "content": user_content},
    ]

AI Execution and Code Synthesis

The AgenticGenerationStage spins up an Agent for each selected Gemini variant. The agent sends the constructed prompt to Gemini via the OpenAI-compatible ChatCompletion endpoint. Gemini parses the video, extracts UI states frame-by-frame, and returns a code block (HTML, CSS, JS). The backend streams the generated chunks back over the WebSocket using "setCode" messages.

# Inside AgenticGenerationStage._run_variant (simplified)

completion = await runner.run(model, prompt_messages)  # model is a Gemini LLM

await self.send_message("setCode", completion, index, None, None)

Why Gemini is Required for Video Processing

Gemini is currently the only provider in the codebase that supports video URLs as image content within the ChatCompletion schema. The selection logic (if input_mode == "video": … return list(VIDEO_VARIANT_MODELS)) guarantees that both variant slots use Gemini models. This limitation exists because Gemini’s multimodal architecture can natively process video sequences, while other LLM providers in the stack lack native video input capabilities.

What the AI Does with the Video Input

When Gemini receives the video data-URL, it performs four distinct analytical steps to generate the prototype:

• Frame analysis – The model extracts visual elements (buttons, text fields, layout grids) from each frame of the recording.
• Interaction inference – By detecting cursor movements and implied click events, the model deduces user actions (e.g., "click ‘Add Item’", "type ‘John’").
• State recreation – The model builds a declarative description of UI state changes that must be reproduced in the final code.
• Code synthesis – Using the selected technology stack (React, Vue, or plain HTML/JS), Gemini emits a complete prototype that mirrors the observed behavior.

All processing happens without any additional server-side video processing; the raw data-URL is sent straight to Gemini, which performs the heavy lifting of video parsing and understanding.

Implementation Examples

Frontend Capture and Transmission

The frontend handles video creation and initiates the WebSocket connection:

// ScreenRecorder.tsx – React state management
const [screenRecordingDataUrl, setScreenRecordingDataUrl] = useState<string | null>(null);

// After blob conversion
setScreenRecordingDataUrl(dataUrl);

// Generation trigger
generateCode([screenRecordingDataUrl], "video");

Backend Model Enforcement

The backend strictly validates Gemini availability before processing video:

# backend/routes/generate_code.py

VIDEO_VARIANT_MODELS = {
    "gemini-1.5-flash",
    "gemini-1.5-pro"
}

def _get_variant_models(self, input_mode: str):
    if input_mode == "video":
        if not self.gemini_api_key:
            raise ValueError("Video mode requires a Gemini API key.")
        return list(VIDEO_VARIANT_MODELS)
    # … other modes …

Prompt Assembly for Video Analysis

The video prompt explicitly instructs the AI to treat the input as a dynamic interaction:

# backend/prompts/create/video.py

system_content = """You are an expert frontend developer analyzing a screen recording.
Watch the video carefully to understand the UI flow, transitions, and user interactions."""

user_content = [
    {"type": "image_url", "image_url": {"url": video_data_url}},
    {"type": "text", "text": "Generate a functional prototype that replicates this exact behavior."}
]

Summary

• The video/screen recording mode captures WebM clips via MediaRecorder and converts them to Base64 data-URLs for transmission.
• The WebSocket pipeline in generateCode.ts streams the video with inputMode: "video" to trigger specialized backend handling.
• Gemini 1.5 models are exclusively required for video processing, enforced by VIDEO_VARIANT_MODELS selection logic.
• The prompt builder embeds the video as an image_url type and instructs the model to analyze frames, infer interactions, and recreate UI states.
• No server-side video processing occurs; Gemini handles all video parsing, frame extraction, and code synthesis directly from the data-URL.

Frequently Asked Questions

Why does video mode require a Gemini API key specifically?

Video mode requires a Gemini API key because Gemini is currently the only model provider in the codebase that supports video URLs as multimodal input within the ChatCompletion schema. The ModelSelectionStage explicitly checks for GEMINI_API_KEY and raises an exception if missing, as other LLM providers cannot process video content natively.

What video format does the screen recorder produce?

The ScreenRecorder component produces WebM format video with MIME type video/webm. The implementation in frontend/src/components/recording/ScreenRecorder.tsx uses the fixWebmDuration utility to ensure the Blob has correct metadata before converting to a Base64 data-URL for transmission.

Is the video processed on the server before being sent to the AI?

No. The video undergoes zero server-side processing. The frontend sends the raw Base64 data-URL directly through the WebSocket to the backend, which immediately forwards it to Gemini's API within the prompt context. All video parsing, frame analysis, and interaction inference happens within Gemini's multimodal processing pipeline.

Can the AI detect specific user interactions like clicking and typing?

Yes. According to the prompt instructions in backend/prompts/create/video.py, Gemini analyzes the video to detect cursor movements, click events, and text input patterns. The model infers user actions such as "click 'Add Item'" or "type 'John'" and recreates these interactive behaviors in the generated HTML/JS prototype code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}