How System Prompts Are Structured for Code Generation Tasks in Screenshot-to-Code

The repository uses a single, reusable system prompt that defines the agent’s role, tooling constraints, and stack-specific requirements, injecting it as the first message in every OpenAI chat completion while varying only the user content based on the input modality.

The abi/screenshot-to-code repository generates UI code from text, images, and video using a consistent prompt engineering strategy. Understanding how system prompts are structured for code generation tasks reveals an architecture that separates agent identity from task-specific instructions, enabling maintainable multi-modal generation pipelines.

Core System Prompt Architecture

All generation tasks rely on a centralized system prompt defined in backend/prompts/system_prompt.py. This file contains a multi-section string spanning lines 1–87 that establishes the foundation for every LLM interaction.

Prompt Sections and Purpose

The SYSTEM_PROMPT constant is organized into four distinct sections that govern agent behavior:

• Tone and style: Mandates concise replies, prohibits raw code in chat responses, and enforces language-aware output formatting.
• Tooling instructions: Documents available file-creation and editing tools, image generation capabilities, background removal features, and the convention of generating a single index.html file.
• Stack-specific instructions: Provides CDN includes and framework-specific usage rules for Tailwind, HTML/CSS, Bootstrap, React, Ionic, and Vue.
• General instructions: Contains font and icon recommendations applicable across all output types.

This prompt is imported throughout the backend as system_prompt.SYSTEM_PROMPT and inserted as the first message in every chat completion request.

Task-Specific Prompt Composition

While the system prompt remains constant, the user prompt varies by input modality. All tasks build a list of ChatCompletionMessageParam objects following this structure:

[
  {"role": "system", "content": "<SYSTEM_PROMPT>"},
  {"role": "user", "content": "<USER_PROMPT_OR_CONTENT>"}
]

Text-to-UI Generation

For text-based requests, the user prompt combines a textual description with stack policies. In backend/prompts/create/text.py (lines 28–33), the build_text_prompt_messages function constructs a plain string containing the text_prompt and selected stack configuration.

Image-to-HTML Generation

Image-based workflows require multipart payloads. The build_image_prompt_messages function in backend/prompts/create/image.py (lines 55–62) sends each screenshot as an image_url part followed by textual instructions, enabling the LLM to analyze visual layouts directly.

Video-to-Interactive-App Generation

Video processing follows a similar pattern to images but with media-specific handling. As implemented in backend/prompts/create/video.py (lines 46–53), the prompt includes a video URL alongside detailed video-analysis instructions in the textual component.

Update from Conversation History

When updating existing projects, the system recreates the full conversation history. The build_update_prompt_from_history function in backend/prompts/update/from_history.py (lines 27–30) prefixes the first user message with stack and image policies while maintaining the dialogue context for iterative refinements.

Design Philosophy and Maintainability

The architecture separates agent identity from task execution through three key principles:

• Consistency: Every generation path enforces the same high-level constraints (tool usage, output formatting, conciseness) by referencing the immutable system prompt.
• Modularity: Stack-specific snippets via build_selected_stack_policy and image policies via build_user_image_policy are injected only into the user portion, keeping the system prompt focused on capabilities rather than specific implementation details.
• Maintainability: Updating global rules—such as adding a new CDN or modifying tool descriptions—requires editing only system_prompt.py, with changes propagating automatically to text, image, video, and update workflows.

Practical Implementation Examples

Building Text-Only UI Prompts

from prompts.create.text import build_text_prompt_messages

messages = build_text_prompt_messages(
    text_prompt="a responsive dashboard with charts",
    stack="Tailwind",
    image_generation_enabled=False,
)

# messages[0] contains the system prompt, messages[1] the user description

Building Image-Based Prompts

from prompts.create.image import build_image_prompt_messages

messages = build_image_prompt_messages(
    image_data_urls=["data:image/png;base64,..."],
    stack="Bootstrap",
    text_prompt="Add a dark mode toggle",
    image_generation_enabled=True,
)

# messages[1]["content"] is a list of multipart parts (image + text)

Updating Projects from History

from prompts.update.from_history import build_update_prompt_from_history

history = [
    {"role": "assistant", "text": "Created index.html"},
    {"role": "user", "text": "Change the header colour", "images": []},
]

messages = build_update_prompt_from_history(
    stack="React",
    history=history,
    image_generation_enabled=False,
)

# The first user message is automatically prefixed with stack & image policy

Summary

• The repository maintains a single system prompt in backend/prompts/system_prompt.py that defines agent behavior, tooling, and stack requirements.
• All generation tasks (text, image, video, and history updates) inject this prompt as the first message in OpenAI chat completions.
• Task-specific variations are handled in the user message through dedicated builder functions: build_text_prompt_messages, build_image_prompt_messages, build_video_prompt_messages, and build_update_prompt_from_history.
• The architecture ensures consistency across modalities while allowing modular injection of stack policies and image generation settings into user prompts only.
• This separation of concerns means updating global instructions requires modifying only one file, with changes propagating to all code generation paths automatically.

Frequently Asked Questions

Where is the main system prompt defined in the screenshot-to-code repository?

The main system prompt is defined in backend/prompts/system_prompt.py within the SYSTEM_PROMPT constant (lines 1–87). This file contains the complete agent instructions covering tone, available tools, stack-specific CDN references, and general formatting guidelines.

How does the system handle different input types like text, images, and video?

Each input type uses a specialized builder function that constructs the user message portion differently while keeping the system prompt constant. Text inputs use build_text_prompt_messages in backend/prompts/create/text.py, images use multipart payloads in backend/prompts/create/image.py, and video uses similar multipart structures in backend/prompts/create/video.py with video-specific analysis instructions.

Why does the codebase use a single system prompt rather than task-specific prompts?

Using a single system prompt ensures consistency in agent behavior across all modalities while reducing maintenance overhead. Stack-specific and image-generation policies are dynamically injected into the user message through helper functions like build_selected_stack_policy, keeping the system prompt focused on core capabilities and identity rather than implementation details.

How are updates to existing projects handled in the prompt structure?

Updates use build_update_prompt_from_history in backend/prompts/update/from_history.py, which recreates the entire conversation history while injecting stack and image policies into the first user message. This approach maintains context for iterative refinements without requiring a separate system prompt configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}