# How Video/Screen Recording Mode Utilizes AI for Prototype Generation in Screenshot-to-Code

> Discover how abi/screenshot-to-code's video recording mode uses AI to generate HTML/JS prototypes from screen recordings. Learn how Gemini 1.5 analyzes visuals and interactions.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The video/screen recording mode captures a WebM clip via the browser's MediaRecorder API, converts it to a Base64 data-URL, and streams it through a WebSocket pipeline where Gemini 1.5 models analyze the visual frames and user interactions to synthesize a functional HTML/JS prototype.**

The `abi/screenshot-to-code` repository enables developers to convert visual inputs into working frontend code. When utilizing the video/screen recording mode for AI prototype generation, the application leverages Google’s Gemini 1.5 models to process dynamic UI interactions frame-by-frame, creating interactive prototypes from screen recordings without any server-side video processing.

## End-to-End AI Pipeline for Video Input

The implementation follows a strict pipeline from browser capture to AI-generated code, with each stage handling specific transformations of the video data.

### Screen Capture and Data-URL Conversion

When a user initiates a recording, the `ScreenRecorder` component in [`frontend/src/components/recording/ScreenRecorder.tsx`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/recording/ScreenRecorder.tsx) instantiates a `MediaRecorder` to collect video blobs. After the user stops recording, the component fixes the WebM duration and converts the final Blob to a Base64 data-URL using `blobToBase64DataUrl`.

```tsx
// ScreenRecorder.tsx – after the recording stops
mediaRecorder.onstop = async () => {
  const completeBlob = await fixWebmDuration(
    new Blob(chunks, { type: "video/webm" })
  );
  const dataUrl = await blobToBase64DataUrl(completeBlob);
  setScreenRecordingDataUrl(dataUrl);
  setScreenRecorderState(ScreenRecorderState.FINISHED);
};

// Kick‑off generation
const kickoffGeneration = () => {
  if (screenRecordingDataUrl) {
    generateCode([screenRecordingDataUrl], "video");
  } else {
    toast.error("Screen recording does not exist.");
  }
};

```

### WebSocket Transmission with Input Mode

The `generateCode` function in [`frontend/src/generateCode.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/generateCode.ts) opens a WebSocket connection to `WS_BACKEND_URL/generate-code` and transmits the video data-URL with `inputMode` explicitly set to `"video"`. This flag triggers the backend’s video-specific processing logic.

```ts
export function generateCode(
  wsRef: React.MutableRefObject<WebSocket | null>,
  params: FullGenerationSettings,
  callbacks: CodeGenerationCallbacks
) {
  const ws = new WebSocket(`${WS_BACKEND_URL}/generate-code`);
  wsRef.current = ws;

  ws.addEventListener("open", () => ws.send(JSON.stringify(params)));
  // … handle streaming messages …
}

```

The `params` object includes:

```json
{
  "inputMode": "video",
  "prompt": { "videos": ["data:video/webm;base64,…"] },
  "stack": "react"
}

```

### Gemini-Only Model Selection

Upon receiving the WebSocket request, the backend forces the use of Gemini models exclusively. In [`backend/routes/generate_code.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py), the `ModelSelectionStage._get_variant_models` method detects `input_mode == "video"` and returns only `VIDEO_VARIANT_MODELS`, which are Gemini 1.5 configurations. If no `GEMINI_API_KEY` is present, the request fails immediately.

```python

# backend/routes/generate_code.py – Model selection

if input_mode == "video":
    if not gemini_api_key:
        raise Exception("Video mode requires a Gemini API key.")
    return list(VIDEO_VARIANT_MODELS)   # two Gemini variants

```

This design enables a side-by-side comparison of two different Gemini configurations (minimal and high-quality) for the same video input.

### Video-Aware Prompt Construction

The prompt builder in [`backend/prompts/create/__init__.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/__init__.py) routes video requests to `build_video_prompt_messages` in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py). This function constructs a **ChatCompletion** message array where the video data-URL is embedded as an `image_url` content part—a format that Gemini treats as a video URL. The system prompt instructs the model to watch the entire video, replicate the UI, and make it functional.

```python
def build_video_prompt_messages(
    video_data_url: str,
    stack: Stack,
    text_prompt: str,
    image_generation_enabled: bool,
) -> list[ChatCompletionMessageParam]:
    user_content = [
        {"type": "image_url", "image_url": {"url": video_data_url, "detail": "high"}},
        {"type": "text", "text": f"""
You have been given a video of a user interacting with a web app...
{selected_stack}
""" + ("\nAdditional instructions: " + text_prompt if text_prompt.strip() else ""))}
    ]
    return [
        {"role": "system", "content": system_prompt.SYSTEM_PROMPT},
        {"role": "user",   "content": user_content},
    ]

```

### AI Execution and Code Synthesis

The `AgenticGenerationStage` spins up an `Agent` for each selected Gemini variant. The agent sends the constructed prompt to Gemini via the OpenAI-compatible `ChatCompletion` endpoint. Gemini parses the video, extracts UI states frame-by-frame, and returns a code block (HTML, CSS, JS). The backend streams the generated chunks back over the WebSocket using `"setCode"` messages.

```python

# Inside AgenticGenerationStage._run_variant (simplified)

completion = await runner.run(model, prompt_messages)  # model is a Gemini LLM

await self.send_message("setCode", completion, index, None, None)

```

## Why Gemini is Required for Video Processing

Gemini is currently the only provider in the codebase that supports **video URLs** as image content within the ChatCompletion schema. The selection logic (`if input_mode == "video": … return list(VIDEO_VARIANT_MODELS)`) guarantees that both variant slots use Gemini models. This limitation exists because Gemini’s multimodal architecture can natively process video sequences, while other LLM providers in the stack lack native video input capabilities.

## What the AI Does with the Video Input

When Gemini receives the video data-URL, it performs four distinct analytical steps to generate the prototype:

- **Frame analysis** – The model extracts visual elements (buttons, text fields, layout grids) from each frame of the recording.
- **Interaction inference** – By detecting cursor movements and implied click events, the model deduces user actions (e.g., "click ‘Add Item’", "type ‘John’").
- **State recreation** – The model builds a declarative description of UI state changes that must be reproduced in the final code.
- **Code synthesis** – Using the selected technology stack (React, Vue, or plain HTML/JS), Gemini emits a complete prototype that mirrors the observed behavior.

All processing happens **without any additional server-side video processing**; the raw data-URL is sent straight to Gemini, which performs the heavy lifting of video parsing and understanding.

## Implementation Examples

### Frontend Capture and Transmission

The frontend handles video creation and initiates the WebSocket connection:

```tsx
// ScreenRecorder.tsx – React state management
const [screenRecordingDataUrl, setScreenRecordingDataUrl] = useState<string | null>(null);

// After blob conversion
setScreenRecordingDataUrl(dataUrl);

// Generation trigger
generateCode([screenRecordingDataUrl], "video");

```

### Backend Model Enforcement

The backend strictly validates Gemini availability before processing video:

```python

# backend/routes/generate_code.py

VIDEO_VARIANT_MODELS = {
    "gemini-1.5-flash",
    "gemini-1.5-pro"
}

def _get_variant_models(self, input_mode: str):
    if input_mode == "video":
        if not self.gemini_api_key:
            raise ValueError("Video mode requires a Gemini API key.")
        return list(VIDEO_VARIANT_MODELS)
    # … other modes …

```

### Prompt Assembly for Video Analysis

The video prompt explicitly instructs the AI to treat the input as a dynamic interaction:

```python

# backend/prompts/create/video.py

system_content = """You are an expert frontend developer analyzing a screen recording.
Watch the video carefully to understand the UI flow, transitions, and user interactions."""

user_content = [
    {"type": "image_url", "image_url": {"url": video_data_url}},
    {"type": "text", "text": "Generate a functional prototype that replicates this exact behavior."}
]

```

## Summary

- The **video/screen recording mode** captures WebM clips via `MediaRecorder` and converts them to Base64 data-URLs for transmission.
- The **WebSocket pipeline** in [`generateCode.ts`](https://github.com/abi/screenshot-to-code/blob/main/generateCode.ts) streams the video with `inputMode: "video"` to trigger specialized backend handling.
- **Gemini 1.5 models** are exclusively required for video processing, enforced by `VIDEO_VARIANT_MODELS` selection logic.
- The **prompt builder** embeds the video as an `image_url` type and instructs the model to analyze frames, infer interactions, and recreate UI states.
- **No server-side video processing** occurs; Gemini handles all video parsing, frame extraction, and code synthesis directly from the data-URL.

## Frequently Asked Questions

### Why does video mode require a Gemini API key specifically?

Video mode requires a Gemini API key because Gemini is currently the only model provider in the codebase that supports video URLs as multimodal input within the ChatCompletion schema. The `ModelSelectionStage` explicitly checks for `GEMINI_API_KEY` and raises an exception if missing, as other LLM providers cannot process video content natively.

### What video format does the screen recorder produce?

The `ScreenRecorder` component produces **WebM** format video with MIME type `video/webm`. The implementation in [`frontend/src/components/recording/ScreenRecorder.tsx`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/recording/ScreenRecorder.tsx) uses the `fixWebmDuration` utility to ensure the Blob has correct metadata before converting to a Base64 data-URL for transmission.

### Is the video processed on the server before being sent to the AI?

No. The video undergoes **zero server-side processing**. The frontend sends the raw Base64 data-URL directly through the WebSocket to the backend, which immediately forwards it to Gemini's API within the prompt context. All video parsing, frame analysis, and interaction inference happens within Gemini's multimodal processing pipeline.

### Can the AI detect specific user interactions like clicking and typing?

Yes. According to the prompt instructions in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py), Gemini analyzes the video to detect **cursor movements, click events, and text input patterns**. The model infers user actions such as "click 'Add Item'" or "type 'John'" and recreates these interactive behaviors in the generated HTML/JS prototype code.