How the Screenshot-to-Code Prompt Pipeline Structures LLM Messages for Images, Text, and Video

TLDR: The abi/screenshot-to-code prompt pipeline uses an asynchronous orchestrator in backend/prompts/pipeline.py to select a construction strategy, then delegates to mode-specific builders in backend/prompts/create/ that format system and user messages for image, text, or video inputs.

The prompt pipeline in the abi/screenshot-to-code repository converts multi-modal user inputs—screenshots, text descriptions, or video recordings—into structured LLM message arrays. It dynamically selects between creation and update strategies while applying stack-specific policies to generate the final prompt. This article traces the pipeline's architecture from request entry to mode-specific message formatting.

Entry Point and Strategy Selection in pipeline.py

The pipeline's primary entry point is the asynchronous function build_prompt_messages located in backend/prompts/pipeline.py. This function accepts several key parameters that determine how the final prompt is constructed:

• stack: The target UI stack configuration (e.g., Tailwind CSS, Material-UI)
• input_mode: One of "image", "text", or "video" (defined in backend/custom_types.py)
• generation_type: Either "create" or "update"
• prompt: The user payload containing text, images, or videos
• history: Previous LLM messages for update workflows
• file_state: Optional filesystem snapshot for update-from-snapshot scenarios
• image_generation_enabled: Feature flag for image generation policies

(See lines 44‑49 in pipeline.py)

First, the pipeline derives a construction plan via derive_prompt_construction_plan, which selects one of three strategies:

1. "create": For new UI generation (default)
2. "update_from_history": For iterative updates based on chat history
3. "update_from_file_snapshot": For updates using current filesystem state

When the strategy resolves to creation, the pipeline delegates to build_create_prompt_from_input, passing the input_mode to select the appropriate builder.

return build_create_prompt_from_input(
    input_mode,
    stack,
    prompt,
    image_generation_enabled,
)

Mode-Specific Message Builders

All creation logic resides under backend/prompts/create/ and is dispatched through backend/prompts/create/__init__.py. The dispatcher inspects the input_mode parameter and routes to the specialized builder for that media type.

(See lines 15‑41 in backend/prompts/create/__init__.py)

Text Mode (text.py)

For text inputs, the builder in backend/prompts/create/text.py constructs a simple two-message prompt. It injects the standard system prompt (system_prompt.SYSTEM_PROMPT) and formats the user's text request with stack-specific policy strings.

The function returns a list containing:

• One system message with the base system prompt
• One user message containing the formatted text instruction

(See lines 16‑36 in text.py)

# Conceptual flow for text mode

messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": f"Generate a {stack['ui']} component: {user_text}"}
]

Screenshot (Image) Mode (image.py)

The screenshot builder in backend/prompts/create/image.py handles one or more base64-encoded image URLs. It constructs a multi-part user message where each image is added as a content part with type: "image_url", followed by a text part containing the main instruction and any optional user text.

This builder applies stack selection policies and image handling configurations before returning the combined system and user messages.

(See lines 15‑63 in image.py)

# Structure for image mode

content_parts = [
    {"type": "image_url", "image_url": {"url": "data:image/png;base64,..."}},
    {"type": "text", "text": "Create a React component matching this design"}
]
messages = [
    {"role": "system", "content": SYSTEM_PROMPT},
    {"role": "user", "content": content_parts}
]

Video Mode (video.py)

The video builder in backend/prompts/create/video.py processes single video inputs. Similar to image mode, it constructs a multi-part message, but includes specific instructions in the text portion describing how to analyze motion, interactions, and styling from the video content.

Notably, the video is transmitted as an "image_url" content part because the OpenAI API treats video content as high-resolution images for multimodal processing.

(See lines 6‑55 in video.py)

# Structure for video mode (video sent as image_url type)

content_parts = [
    {"type": "image_url", "image_url": {"url": "data:video/mp4;base64,..."}},
    {"type": "text", "text": "Analyze this video and recreate the UI interactions..."}
]

Type Definitions and Supporting Infrastructure

The pipeline relies on strict type definitions in backend/custom_types.py, where InputMode is defined as a Literal type:

InputMode = Literal["image", "text", "video"]

Additional supporting files include:

• backend/prompts/prompt_types.py: Defines core types like Stack and UserTurnInput
• backend/prompts/message_builder.py: Provides the Prompt type alias for ChatCompletionMessageParam arrays
• backend/llm.py: The LLM client that ultimately transmits the formatted messages

End-to-End Usage Example

The following example demonstrates how to invoke the prompt pipeline for all three input modes:

from prompts.pipeline import build_prompt_messages
from prompts.prompt_types import Stack
from custom_types import InputMode

stack: Stack = {"framework": "React", "ui": "Tailwind"}

# 1. Text mode

txt_prompt = {"text": "Create a landing page with a hero section"}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="text",
    generation_type="create",
    prompt=txt_prompt,
    history=[],
)

# 2. Screenshot (image) mode

img_prompt = {
    "text": "Add a dark header",
    "images": ["data:image/png;base64,iVBORw0KGgoAAAANS..."]
}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="image",
    generation_type="create",
    prompt=img_prompt,
    history=[],
)

# 3. Video mode

video_prompt = {
    "text": "",
    "videos": ["data:video/mp4;base64,AAAAHGZ0eXB..."]
}
messages = await build_prompt_messages(
    stack=stack,
    input_mode="video",
    generation_type="create",
    prompt=video_prompt,
    history=[],
)

Summary

• The prompt pipeline centers on build_prompt_messages in backend/prompts/pipeline.py, which orchestrates message construction through strategy selection.
• Three input modes are supported via the InputMode literal: "image" (screenshots), "text", and "video".
• The pipeline selects between creation and update strategies (derive_prompt_construction_plan) before delegating to mode-specific builders.
• Text mode produces simple text-based user messages, while image and video modes generate multi-part content with base64-encoded media.
• All mode-specific builders reside in backend/prompts/create/ and return standardized Prompt arrays consumed by backend/llm.py.

Frequently Asked Questions

What input modes does the screenshot-to-code prompt pipeline support?

The pipeline supports three distinct input modes defined in backend/custom_types.py as the InputMode literal: "image" for screenshots, "text" for natural language descriptions, and "video" for screen recordings or UI demonstrations.

How does the pipeline decide between create and update strategies?

The function derive_prompt_construction_plan analyzes the presence of history and file_state parameters. If neither is provided, it defaults to the "create" strategy. If historical messages exist, it selects "update_from_history"; if a filesystem snapshot is provided, it selects "update_from_file_snapshot".

Why are videos treated as image URLs in the message payload?

According to the implementation in backend/prompts/create/video.py, videos are embedded as "image_url" content parts because the OpenAI API treats video files as high-resolution images for multimodal model consumption. This allows the LLM to analyze video frames using the same vision capabilities used for static screenshots.

Where is the entry point for building prompt messages?

The primary entry point is the build_prompt_messages asynchronous function located in backend/prompts/pipeline.py at lines 44‑49. This function serves as the orchestrator that determines the construction strategy and dispatches to the appropriate mode-specific builder in backend/prompts/create/.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}