# How the Screenshot-to-Code Prompt Pipeline Structures LLM Messages for Images, Text, and Video

> Discover how the screenshot to code prompt pipeline structures LLM messages for diverse inputs like images, text, and video. Learn about its asynchronous orchestrator and mode-specific builders.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**TLDR:** The `abi/screenshot-to-code` **prompt pipeline** uses an asynchronous orchestrator in [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py) to select a construction strategy, then delegates to mode-specific builders in `backend/prompts/create/` that format system and user messages for image, text, or video inputs.

The **prompt pipeline** in the `abi/screenshot-to-code` repository converts multi-modal user inputs—screenshots, text descriptions, or video recordings—into structured LLM message arrays. It dynamically selects between creation and update strategies while applying stack-specific policies to generate the final prompt. This article traces the pipeline's architecture from request entry to mode-specific message formatting.

## Entry Point and Strategy Selection in [`pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/pipeline.py)

The pipeline's primary entry point is the asynchronous function `build_prompt_messages` located in [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py). This function accepts several key parameters that determine how the final prompt is constructed:

- **`stack`**: The target UI stack configuration (e.g., Tailwind CSS, Material-UI)
- **`input_mode`**: One of `"image"`, `"text"`, or `"video"` (defined in [`backend/custom_types.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/custom_types.py))
- **`generation_type`**: Either `"create"` or `"update"` 
- **`prompt`**: The user payload containing `text`, `images`, or `videos`
- **`history`**: Previous LLM messages for update workflows
- **`file_state`**: Optional filesystem snapshot for update-from-snapshot scenarios
- **`image_generation_enabled`**: Feature flag for image generation policies

(See lines 44‑49 in [`pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/pipeline.py))

First, the pipeline derives a **construction plan** via `derive_prompt_construction_plan`, which selects one of three strategies:

1. **`"create"`**: For new UI generation (default)
2. **`"update_from_history"`**: For iterative updates based on chat history
3. **`"update_from_file_snapshot"`**: For updates using current filesystem state

When the strategy resolves to creation, the pipeline delegates to `build_create_prompt_from_input`, passing the `input_mode` to select the appropriate builder.

```python
return build_create_prompt_from_input(
    input_mode,
    stack,
    prompt,
    image_generation_enabled,
)

```

## Mode-Specific Message Builders

All creation logic resides under `backend/prompts/create/` and is dispatched through [`backend/prompts/create/__init__.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/__init__.py). The dispatcher inspects the `input_mode` parameter and routes to the specialized builder for that media type.

(See lines 15‑41 in [`backend/prompts/create/__init__.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/__init__.py))

### Text Mode ([`text.py`](https://github.com/abi/screenshot-to-code/blob/main/text.py))

For **text inputs**, the builder in [`backend/prompts/create/text.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/text.py) constructs a simple two-message prompt. It injects the standard system prompt (`system_prompt.SYSTEM_PROMPT`) and formats the user's text request with stack-specific policy strings.

The function returns a list containing:
- One **system** message with the base system prompt
- One **user** message containing the formatted text instruction

(See lines 16‑36 in [`text.py`](https://github.com/abi/screenshot-to-code/blob/main/text.py))

```python

# Conceptual flow for text mode

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Generate a {stack['ui']} component: {user_text}"}
]

```

### Screenshot (Image) Mode ([`image.py`](https://github.com/abi/screenshot-to-code/blob/main/image.py))

The **screenshot** builder in [`backend/prompts/create/image.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/image.py) handles one or more base64-encoded image URLs. It constructs a **multi-part** user message where each image is added as a content part with `type: "image_url"`, followed by a text part containing the main instruction and any optional user text.

This builder applies stack selection policies and image handling configurations before returning the combined system and user messages.

(See lines 15‑63 in [`image.py`](https://github.com/abi/screenshot-to-code/blob/main/image.py))

```python

# Structure for image mode

content_parts = [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "Create a React component matching this design"}
]
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": content_parts}
]

```

### Video Mode ([`video.py`](https://github.com/abi/screenshot-to-code/blob/main/video.py))

The **video** builder in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py) processes single video inputs. Similar to image mode, it constructs a multi-part message, but includes specific instructions in the text portion describing how to analyze motion, interactions, and styling from the video content.

Notably, the video is transmitted as an `"image_url"` content part because the OpenAI API treats video content as high-resolution images for multimodal processing.

(See lines 6‑55 in [`video.py`](https://github.com/abi/screenshot-to-code/blob/main/video.py))

```python

# Structure for video mode (video sent as image_url type)

content_parts = [
    {"type": "image_url", "image_url": {"url": "data:video/mp4;base64,..."}},
    {"type": "text", "text": "Analyze this video and recreate the UI interactions..."}
]

```

## Type Definitions and Supporting Infrastructure

The pipeline relies on strict type definitions in [`backend/custom_types.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/custom_types.py), where `InputMode` is defined as a `Literal` type:

```python
InputMode = Literal["image", "text", "video"]

```

Additional supporting files include:
- **[`backend/prompts/prompt_types.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/prompt_types.py)**: Defines core types like `Stack` and `UserTurnInput`
- **[`backend/prompts/message_builder.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/message_builder.py)**: Provides the `Prompt` type alias for `ChatCompletionMessageParam` arrays
- **[`backend/llm.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/llm.py)**: The LLM client that ultimately transmits the formatted messages

## End-to-End Usage Example

The following example demonstrates how to invoke the prompt pipeline for all three input modes:

```python
from prompts.pipeline import build_prompt_messages
from prompts.prompt_types import Stack
from custom_types import InputMode

stack: Stack = {"framework": "React", "ui": "Tailwind"}

# 1. Text mode

txt_prompt = {"text": "Create a landing page with a hero section"}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="text",
    generation_type="create",
    prompt=txt_prompt,
    history=[],
)

# 2. Screenshot (image) mode

img_prompt = {
    "text": "Add a dark header",
    "images": ["data:image/png;base64,iVBORw0KGgoAAAANS..."]
}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="image",
    generation_type="create",
    prompt=img_prompt,
    history=[],
)

# 3. Video mode

video_prompt = {
    "text": "",
    "videos": ["data:video/mp4;base64,AAAAHGZ0eXB..."]
}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="video",
    generation_type="create",
    prompt=video_prompt,
    history=[],
)

```

## Summary

- The **prompt pipeline** centers on `build_prompt_messages` in [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py), which orchestrates message construction through strategy selection.
- Three **input modes** are supported via the `InputMode` literal: `"image"` (screenshots), `"text"`, and `"video"`.
- The pipeline selects between **creation** and **update** strategies (`derive_prompt_construction_plan`) before delegating to mode-specific builders.
- **Text mode** produces simple text-based user messages, while **image** and **video** modes generate multi-part content with base64-encoded media.
- All mode-specific builders reside in `backend/prompts/create/` and return standardized `Prompt` arrays consumed by [`backend/llm.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/llm.py).

## Frequently Asked Questions

### What input modes does the screenshot-to-code prompt pipeline support?

The pipeline supports three distinct input modes defined in [`backend/custom_types.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/custom_types.py) as the `InputMode` literal: `"image"` for screenshots, `"text"` for natural language descriptions, and `"video"` for screen recordings or UI demonstrations.

### How does the pipeline decide between create and update strategies?

The function `derive_prompt_construction_plan` analyzes the presence of `history` and `file_state` parameters. If neither is provided, it defaults to the `"create"` strategy. If historical messages exist, it selects `"update_from_history"`; if a filesystem snapshot is provided, it selects `"update_from_file_snapshot"`.

### Why are videos treated as image URLs in the message payload?

According to the implementation in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py), videos are embedded as `"image_url"` content parts because the OpenAI API treats video files as high-resolution images for multimodal model consumption. This allows the LLM to analyze video frames using the same vision capabilities used for static screenshots.

### Where is the entry point for building prompt messages?

The primary entry point is the `build_prompt_messages` asynchronous function located in [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py) at lines 44‑49. This function serves as the orchestrator that determines the construction strategy and dispatches to the appropriate mode-specific builder in `backend/prompts/create/`.