How Prompt History Messages Are Parsed and Reconstructed for Context in Screenshot-to-Code

The screenshot-to-code application stores every UI interaction as variant-history entries containing asset IDs, then reconstructs them into LLM-ready messages by resolving those IDs to inline data-URLs through a three-step pipeline implemented in frontend/src/lib/prompt-history.ts.

When building AI-powered coding tools, maintaining conversation context requires carefully managing media assets alongside text prompts. In the abi/screenshot-to-code repository, the frontend implements a sophisticated parsing and reconstruction system that converts internal history state into API-compatible payloads. This article examines how prompt history messages are parsed and reconstructed for context, tracing the flow from raw user inputs to structured LLM requests.

The Variant-History Storage Model

The UI maintains conversation state using variant-history entries defined in frontend/src/components/commits/types.ts. Each VariantHistoryMessage records the interaction role (user or assistant), the raw text content, and arrays of asset IDs rather than the actual media data.

// Internal representation stored in the UI state
interface VariantHistoryMessage {
  role: "user" | "assistant";
  text: string;
  imageAssetIds: string[];  // References to PromptAsset objects
  videoAssetIds: string[];  // References to PromptAsset objects
}

This ID-based approach decouples the conversation history from the actual media storage, allowing the application to deduplicate assets and lazy-load media only when constructing the final request.

The Three-Step Reconstruction Pipeline

When preparing a request for the code-generation LLM, the system transforms the internal variant history into a plain PromptHistoryMessage[] array. This reconstruction occurs in three distinct steps within frontend/src/lib/prompt-history.ts.

Step 1: Deep Cloning the Raw History

The cloneVariantHistory function creates a deep copy of the history array to prevent subsequent mutations from affecting the original state. This utility duplicates both the message objects and their asset ID arrays.

export function cloneVariantHistory(history: VariantHistoryMessage[]): VariantHistoryMessage[] {
  return history.map(message => ({
    ...message,
    imageAssetIds: [...message.imageAssetIds],
    videoAssetIds: [...message.videoAssetIds],
  }));
}

Source: frontend/src/lib/prompt-history.ts lines 18-26

Step 2: Resolving Asset IDs to Data URLs

The resolveAssetIdsToDataUrls function bridges the ID-based storage with the LLM's requirement for inline media. It accepts a getAssetsById accessor function that returns the in-memory store mapping IDs to PromptAsset objects ({id, type, dataUrl}).

export function resolveAssetIdsToDataUrls(
  assetIds: string[], 
  getAssetsById: GetAssetsById
): string[] {
  const assetsById = getAssetsById();
  return assetIds
    .map(assetId => assetsById[assetId]?.dataUrl)
    .filter((value): value is string => Boolean(value));
}

Source: frontend/src/lib/prompt-history.ts lines 65-73

This function filters out any missing or undefined assets, ensuring the LLM receives only valid data-URLs.

Step 3: Building the LLM Request Payload

The toRequestHistory function orchestrates the final transformation. It walks the cloned history, swaps ID arrays for resolved data-URL arrays, and returns the payload shape expected by the LLM client.

export function toRequestHistory(
  history: VariantHistoryMessage[],
  getAssetsById: GetAssetsById
): PromptHistoryMessage[] {
  return history.map(message => ({
    role: message.role,
    text: message.text,
    images: resolveAssetIdsToDataUrls(message.imageAssetIds, getAssetsById),
    videos: resolveAssetIdsToDataUrls(message.videoAssetIds, getAssetsById),
  }));
}

Source: frontend/src/lib/prompt-history.ts lines 75-85

Asset Registration and ID Management

Before reconstruction can occur, incoming media must be registered and assigned IDs. The registerAssetIds function handles deduplication against existing assets, generates fresh IDs for new data-URLs using a provided generator function, and persists additions via upsertPromptAssets.

export function registerAssetIds(
  type: AssetType,
  dataUrls: string[],
  getAssetsById: GetAssetsById,
  upsertPromptAssets: (assets: PromptAsset[]) => void,
  generateId: () => string
): string[] {
  // Deduplicates, creates new PromptAsset objects, and returns stable IDs
}

Source: frontend/src/lib/prompt-history.ts lines 28-63

This registration step ensures that identical screenshots or videos reused across multiple messages maintain the same ID, reducing memory overhead and simplifying history management.

Complete Implementation Example

The following workflow demonstrates the full round-trip from user input to LLM-ready payload:

import { 
  buildUserHistoryMessage, 
  registerAssetIds, 
  toRequestHistory 
} from "./prompt-history";

// 1. Create a message placeholder
const userMsg = buildUserHistoryMessage(
  "Create a responsive navbar with this design",
  [],  // image IDs populated after registration
  []
);

// 2. Register dropped file data-URLs
const imageIds = registerAssetIds(
  "image",
  ["data:image/png;base64,iVBORw0KGgo..."],
  () => assetsById,
  newAssets => persistToStore(newAssets),
  () => `asset-${crypto.randomUUID()}`
);

// 3. Attach IDs to the message
userMsg.imageAssetIds = imageIds;

// 4. Reconstruct for the LLM API
const payload = toRequestHistory([userMsg, ...previousHistory], () => assetsById);

// Result: Array of messages with inline data-URLs ready for OpenAI/Claude APIs

Summary

• Variant-history entries store conversation context as ID references in frontend/src/components/commits/types.ts, separating metadata from media data.
• Deep cloning via cloneVariantHistory protects immutable history state during transformation.
• Asset resolution through resolveAssetIdsToDataUrls converts stored IDs to inline data-URLs while filtering invalid references.
• Payload construction in toRequestHistory produces the final PromptHistoryMessage[] structure required by LLM clients.
• Registration logic in registerAssetIds handles deduplication and ID generation for new media assets.

Frequently Asked Questions

What data structure does the UI use to store prompt history internally?

The UI uses VariantHistoryMessage objects defined in frontend/src/components/commits/types.ts. These entries contain the message role (user or assistant), text content, and arrays of asset IDs (imageAssetIds and videoAssetIds) rather than the actual media data. This ID-based storage enables efficient deduplication and separates conversation state from binary asset storage.

How does the application handle missing or invalid asset IDs during reconstruction?

During the reconstruction phase, resolveAssetIdsToDataUrls filters out any undefined or missing assets using a Boolean type guard. If an ID no longer exists in the asset store, it is silently dropped from the resulting data-URL array, ensuring the LLM receives only valid media references without throwing runtime errors.

Why does the reconstruction process use asset IDs instead of storing data URLs directly?

Storing asset IDs rather than inline data-URLs provides three advantages: deduplication of identical media across multiple messages, reduced memory footprint for the React state tree, and flexible asset management allowing updates to media content without rewriting entire conversation histories. The IDs are resolved to full data-URLs only at the moment of API request construction.

Where is the prompt history reconstruction logic tested?

The unit tests in frontend/src/lib/prompt-history.test.ts verify the complete reconstruction pipeline. These tests confirm that cloneVariantHistory properly deep-copies nested arrays, that registerAssetIds correctly deduplicates existing assets while upserting new ones, and that toRequestHistory accurately resolves IDs to data-URLs in the final payload shape.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}