# How Prompt History Messages Are Parsed and Reconstructed for Context in Screenshot-to-Code

> Discover how screenshot-to-code reconstructs prompt history messages for context. Learn about its three-step pipeline for parsing UI interactions and generating LLM-ready messages.

- Repository: [Abi Raja/screenshot-to-code](https://github.com/abi/screenshot-to-code)
- Tags: internals
- Published: 2026-03-02

---

**The screenshot-to-code application stores every UI interaction as variant-history entries containing asset IDs, then reconstructs them into LLM-ready messages by resolving those IDs to inline data-URLs through a three-step pipeline implemented in [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts).**

When building AI-powered coding tools, maintaining conversation context requires carefully managing media assets alongside text prompts. In the `abi/screenshot-to-code` repository, the frontend implements a sophisticated parsing and reconstruction system that converts internal history state into API-compatible payloads. This article examines how prompt history messages are parsed and reconstructed for context, tracing the flow from raw user inputs to structured LLM requests.

## The Variant-History Storage Model

The UI maintains conversation state using **variant-history entries** defined in [`frontend/src/components/commits/types.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/commits/types.ts). Each `VariantHistoryMessage` records the interaction role (`user` or `assistant`), the raw text content, and arrays of asset IDs rather than the actual media data.

```ts
// Internal representation stored in the UI state
interface VariantHistoryMessage {
  role: "user" | "assistant";
  text: string;
  imageAssetIds: string[];  // References to PromptAsset objects
  videoAssetIds: string[];  // References to PromptAsset objects
}

```

This ID-based approach decouples the conversation history from the actual media storage, allowing the application to deduplicate assets and lazy-load media only when constructing the final request.

## The Three-Step Reconstruction Pipeline

When preparing a request for the code-generation LLM, the system transforms the internal variant history into a plain `PromptHistoryMessage[]` array. This reconstruction occurs in three distinct steps within [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts).

### Step 1: Deep Cloning the Raw History

The `cloneVariantHistory` function creates a deep copy of the history array to prevent subsequent mutations from affecting the original state. This utility duplicates both the message objects and their asset ID arrays.

```ts
export function cloneVariantHistory(history: VariantHistoryMessage[]): VariantHistoryMessage[] {
  return history.map(message => ({
    ...message,
    imageAssetIds: [...message.imageAssetIds],
    videoAssetIds: [...message.videoAssetIds],
  }));
}

```

*Source: [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts) lines 18-26*

### Step 2: Resolving Asset IDs to Data URLs

The `resolveAssetIdsToDataUrls` function bridges the ID-based storage with the LLM's requirement for inline media. It accepts a `getAssetsById` accessor function that returns the in-memory store mapping IDs to `PromptAsset` objects (`{id, type, dataUrl}`).

```ts
export function resolveAssetIdsToDataUrls(
  assetIds: string[], 
  getAssetsById: GetAssetsById
): string[] {
  const assetsById = getAssetsById();
  return assetIds
    .map(assetId => assetsById[assetId]?.dataUrl)
    .filter((value): value is string => Boolean(value));
}

```

*Source: [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts) lines 65-73*

This function filters out any missing or undefined assets, ensuring the LLM receives only valid data-URLs.

### Step 3: Building the LLM Request Payload

The `toRequestHistory` function orchestrates the final transformation. It walks the cloned history, swaps ID arrays for resolved data-URL arrays, and returns the payload shape expected by the LLM client.

```ts
export function toRequestHistory(
  history: VariantHistoryMessage[],
  getAssetsById: GetAssetsById
): PromptHistoryMessage[] {
  return history.map(message => ({
    role: message.role,
    text: message.text,
    images: resolveAssetIdsToDataUrls(message.imageAssetIds, getAssetsById),
    videos: resolveAssetIdsToDataUrls(message.videoAssetIds, getAssetsById),
  }));
}

```

*Source: [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts) lines 75-85*

## Asset Registration and ID Management

Before reconstruction can occur, incoming media must be registered and assigned IDs. The `registerAssetIds` function handles deduplication against existing assets, generates fresh IDs for new data-URLs using a provided generator function, and persists additions via `upsertPromptAssets`.

```ts
export function registerAssetIds(
  type: AssetType,
  dataUrls: string[],
  getAssetsById: GetAssetsById,
  upsertPromptAssets: (assets: PromptAsset[]) => void,
  generateId: () => string
): string[] {
  // Deduplicates, creates new PromptAsset objects, and returns stable IDs
}

```

*Source: [`frontend/src/lib/prompt-history.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.ts) lines 28-63*

This registration step ensures that identical screenshots or videos reused across multiple messages maintain the same ID, reducing memory overhead and simplifying history management.

## Complete Implementation Example

The following workflow demonstrates the full round-trip from user input to LLM-ready payload:

```ts
import { 
  buildUserHistoryMessage, 
  registerAssetIds, 
  toRequestHistory 
} from "./prompt-history";

// 1. Create a message placeholder
const userMsg = buildUserHistoryMessage(
  "Create a responsive navbar with this design",
  [],  // image IDs populated after registration
  []
);

// 2. Register dropped file data-URLs
const imageIds = registerAssetIds(
  "image",
  ["data:image/png;base64,iVBORw0KGgo..."],
  () => assetsById,
  newAssets => persistToStore(newAssets),
  () => `asset-${crypto.randomUUID()}`
);

// 3. Attach IDs to the message
userMsg.imageAssetIds = imageIds;

// 4. Reconstruct for the LLM API
const payload = toRequestHistory([userMsg, ...previousHistory], () => assetsById);

// Result: Array of messages with inline data-URLs ready for OpenAI/Claude APIs

```

## Summary

- **Variant-history entries** store conversation context as ID references in [`frontend/src/components/commits/types.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/commits/types.ts), separating metadata from media data.
- **Deep cloning** via `cloneVariantHistory` protects immutable history state during transformation.
- **Asset resolution** through `resolveAssetIdsToDataUrls` converts stored IDs to inline data-URLs while filtering invalid references.
- **Payload construction** in `toRequestHistory` produces the final `PromptHistoryMessage[]` structure required by LLM clients.
- **Registration logic** in `registerAssetIds` handles deduplication and ID generation for new media assets.

## Frequently Asked Questions

### What data structure does the UI use to store prompt history internally?

The UI uses `VariantHistoryMessage` objects defined in [`frontend/src/components/commits/types.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/commits/types.ts). These entries contain the message role (`user` or `assistant`), text content, and arrays of asset IDs (`imageAssetIds` and `videoAssetIds`) rather than the actual media data. This ID-based storage enables efficient deduplication and separates conversation state from binary asset storage.

### How does the application handle missing or invalid asset IDs during reconstruction?

During the reconstruction phase, `resolveAssetIdsToDataUrls` filters out any undefined or missing assets using a Boolean type guard. If an ID no longer exists in the asset store, it is silently dropped from the resulting data-URL array, ensuring the LLM receives only valid media references without throwing runtime errors.

### Why does the reconstruction process use asset IDs instead of storing data URLs directly?

Storing asset IDs rather than inline data-URLs provides three advantages: **deduplication** of identical media across multiple messages, **reduced memory footprint** for the React state tree, and **flexible asset management** allowing updates to media content without rewriting entire conversation histories. The IDs are resolved to full data-URLs only at the moment of API request construction.

### Where is the prompt history reconstruction logic tested?

The unit tests in [`frontend/src/lib/prompt-history.test.ts`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/lib/prompt-history.test.ts) verify the complete reconstruction pipeline. These tests confirm that `cloneVariantHistory` properly deep-copies nested arrays, that `registerAssetIds` correctly deduplicates existing assets while upserting new ones, and that `toRequestHistory` accurately resolves IDs to data-URLs in the final payload shape.