# How System Prompts Are Structured for Code Generation Tasks in Screenshot-to-Code

> Discover how system prompts structure code generation for abi/screenshot-to-code. Learn about reusable prompts defining agent roles, tool constraints, and stack requirements for efficient AI development.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The repository uses a single, reusable system prompt that defines the agent’s role, tooling constraints, and stack-specific requirements, injecting it as the first message in every OpenAI chat completion while varying only the user content based on the input modality.**

The `abi/screenshot-to-code` repository generates UI code from text, images, and video using a consistent prompt engineering strategy. Understanding how system prompts are structured for code generation tasks reveals an architecture that separates agent identity from task-specific instructions, enabling maintainable multi-modal generation pipelines.

## Core System Prompt Architecture

All generation tasks rely on a centralized system prompt defined in [`backend/prompts/system_prompt.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/system_prompt.py). This file contains a multi-section string spanning lines 1–87 that establishes the foundation for every LLM interaction.

### Prompt Sections and Purpose

The `SYSTEM_PROMPT` constant is organized into four distinct sections that govern agent behavior:

- **Tone and style**: Mandates concise replies, prohibits raw code in chat responses, and enforces language-aware output formatting.
- **Tooling instructions**: Documents available file-creation and editing tools, image generation capabilities, background removal features, and the convention of generating a single [`index.html`](https://github.com/abi/screenshot-to-code/blob/main/index.html) file.
- **Stack-specific instructions**: Provides CDN includes and framework-specific usage rules for Tailwind, HTML/CSS, Bootstrap, React, Ionic, and Vue.
- **General instructions**: Contains font and icon recommendations applicable across all output types.

This prompt is imported throughout the backend as `system_prompt.SYSTEM_PROMPT` and inserted as the first message in every chat completion request.

## Task-Specific Prompt Composition

While the system prompt remains constant, the user prompt varies by input modality. All tasks build a list of `ChatCompletionMessageParam` objects following this structure:

```json
[
  {"role": "system", "content": "<SYSTEM_PROMPT>"},
  {"role": "user", "content": "<USER_PROMPT_OR_CONTENT>"}
]

```

### Text-to-UI Generation

For text-based requests, the user prompt combines a textual description with stack policies. In [`backend/prompts/create/text.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/text.py) (lines 28–33), the `build_text_prompt_messages` function constructs a plain string containing the `text_prompt` and selected stack configuration.

### Image-to-HTML Generation

Image-based workflows require multipart payloads. The `build_image_prompt_messages` function in [`backend/prompts/create/image.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/image.py) (lines 55–62) sends each screenshot as an `image_url` part followed by textual instructions, enabling the LLM to analyze visual layouts directly.

### Video-to-Interactive-App Generation

Video processing follows a similar pattern to images but with media-specific handling. As implemented in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py) (lines 46–53), the prompt includes a video URL alongside detailed video-analysis instructions in the textual component.

### Update from Conversation History

When updating existing projects, the system recreates the full conversation history. The `build_update_prompt_from_history` function in [`backend/prompts/update/from_history.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/update/from_history.py) (lines 27–30) prefixes the first user message with stack and image policies while maintaining the dialogue context for iterative refinements.

## Design Philosophy and Maintainability

The architecture separates **agent identity** from **task execution** through three key principles:

- **Consistency**: Every generation path enforces the same high-level constraints (tool usage, output formatting, conciseness) by referencing the immutable system prompt.
- **Modularity**: Stack-specific snippets via `build_selected_stack_policy` and image policies via `build_user_image_policy` are injected only into the user portion, keeping the system prompt focused on capabilities rather than specific implementation details.
- **Maintainability**: Updating global rules—such as adding a new CDN or modifying tool descriptions—requires editing only [`system_prompt.py`](https://github.com/abi/screenshot-to-code/blob/main/system_prompt.py), with changes propagating automatically to text, image, video, and update workflows.

## Practical Implementation Examples

### Building Text-Only UI Prompts

```python
from prompts.create.text import build_text_prompt_messages

messages = build_text_prompt_messages(
    text_prompt="a responsive dashboard with charts",
    stack="Tailwind",
    image_generation_enabled=False,
)

# messages[0] contains the system prompt, messages[1] the user description

```

### Building Image-Based Prompts

```python
from prompts.create.image import build_image_prompt_messages

messages = build_image_prompt_messages(
    image_data_urls=["data:image/png;base64,..."],
    stack="Bootstrap",
    text_prompt="Add a dark mode toggle",
    image_generation_enabled=True,
)

# messages[1]["content"] is a list of multipart parts (image + text)

```

### Updating Projects from History

```python
from prompts.update.from_history import build_update_prompt_from_history

history = [
    {"role": "assistant", "text": "Created index.html"},
    {"role": "user", "text": "Change the header colour", "images": []},
]

messages = build_update_prompt_from_history(
    stack="React",
    history=history,
    image_generation_enabled=False,
)

# The first user message is automatically prefixed with stack & image policy

```

## Summary

- The repository maintains a **single system prompt** in [`backend/prompts/system_prompt.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/system_prompt.py) that defines agent behavior, tooling, and stack requirements.
- All generation tasks (text, image, video, and history updates) inject this prompt as the first message in OpenAI chat completions.
- **Task-specific variations** are handled in the user message through dedicated builder functions: `build_text_prompt_messages`, `build_image_prompt_messages`, `build_video_prompt_messages`, and `build_update_prompt_from_history`.
- The architecture ensures **consistency** across modalities while allowing **modular** injection of stack policies and image generation settings into user prompts only.
- This separation of concerns means updating global instructions requires modifying only one file, with changes propagating to all code generation paths automatically.

## Frequently Asked Questions

### Where is the main system prompt defined in the screenshot-to-code repository?

The main system prompt is defined in [`backend/prompts/system_prompt.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/system_prompt.py) within the `SYSTEM_PROMPT` constant (lines 1–87). This file contains the complete agent instructions covering tone, available tools, stack-specific CDN references, and general formatting guidelines.

### How does the system handle different input types like text, images, and video?

Each input type uses a specialized builder function that constructs the user message portion differently while keeping the system prompt constant. Text inputs use `build_text_prompt_messages` in [`backend/prompts/create/text.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/text.py), images use multipart payloads in [`backend/prompts/create/image.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/image.py), and video uses similar multipart structures in [`backend/prompts/create/video.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/create/video.py) with video-specific analysis instructions.

### Why does the codebase use a single system prompt rather than task-specific prompts?

Using a single system prompt ensures **consistency** in agent behavior across all modalities while reducing maintenance overhead. Stack-specific and image-generation policies are dynamically injected into the user message through helper functions like `build_selected_stack_policy`, keeping the system prompt focused on core capabilities and identity rather than implementation details.

### How are updates to existing projects handled in the prompt structure?

Updates use `build_update_prompt_from_history` in [`backend/prompts/update/from_history.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/update/from_history.py), which recreates the entire conversation history while injecting stack and image policies into the first user message. This approach maintains context for iterative refinements without requiring a separate system prompt configuration.