# How Pi Web Handles Image Drag-and-Drop with Agent Prompt Integration

> Learn how Pi Web seamlessly integrates image drag-and-drop. See how it converts images to base-64 data URLs and injects them into agent prompts for real-time LLM visual analysis.

- Repository: [Alex Yang/pi-web](https://github.com/agegr/pi-web)
- Tags: how-to-guide
- Published: 2026-07-25

---

**Pi Web converts dropped images into base-64 data URLs and injects them as structured tool calls into the agent's prompt context, allowing the LLM to analyze visual content in real time.**

The `agegr/pi-web` repository implements a seamless image drag-and-drop system that bridges browser file handling with LLM agent capabilities. By leveraging the **`useDragDrop`** hook and the **`ChatWindow`** component, the application transforms raw image files into contextual prompt data that the agent can reference and reason about during conversations.

## The Drag-and-Drop Architecture

### Event Handling with useDragDrop

The core functionality resides in [`hooks/useDragDrop.ts`](https://github.com/agegr/pi-web/blob/main/hooks/useDragDrop.ts), which registers global browser listeners for **`drag-enter`**, **`drag-over`**, and **`drop`** events. When a user drags an image file over the chat interface, the hook sets a reactive state flag that provides immediate visual feedback. On drop, it intercepts the native `DragEvent` and accesses `event.dataTransfer.files` to retrieve the file list.

The hook filters incoming files by MIME type to ensure only image formats (such as `image/png`, `image/jpeg`, or `image/webp`) are processed. This validation occurs client-side before any data conversion begins, preventing invalid file types from reaching the agent pipeline.

### File Processing and Base-64 Conversion

Once validated, the hook instantiates a **`FileReader`** to asynchronously read the binary image data. It converts the file to a base-64 encoded data URL using `readAsDataURL()`, producing a string in the format `data:image/*;base64,…`. This string representation is necessary because the agent prompt system requires text-serializable content.

The hook then invokes a callback—typically provided by the parent component—with two arguments: the base-64 data URL and the original filename. This decouples the file handling logic from the chat state management, following the single-responsibility principle.

## Agent Prompt Integration via Tool Calls

### Tool Call Construction in ChatWindow

The [`components/ChatWindow.tsx`](https://github.com/agegr/pi-web/blob/main/components/ChatWindow.tsx) file orchestrates the integration between the UI and the agent. When `useDragDrop` returns an image payload, `ChatWindow` constructs a specialized **tool call** object that conforms to Pi Web's internal messaging protocol. The payload structure follows this schema:

```json
{
  "type": "toolCall",
  "id": "t1",
  "name": "image",
  "arguments": {
    "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
    "fileName": "snapshot.png"
  }
}

```

`ChatWindow` appends this object to the conversation history as a new message entry. By treating the image as a tool call rather than plain text, the system maintains type safety and allows the backend to apply specific preprocessing logic for visual inputs before they reach the LLM.

### How the Agent Processes Visual Input

When the conversation stream reaches the agent, the backend detects the `image` tool call and extracts the base-64 data. The agent framework converts this data into a format the underlying LLM can interpret—typically by inserting a descriptive marker or processing the image through a vision-capable model endpoint.

The agent then receives the image as part of its prompt context, enabling it to reference specific visual elements in its response. For example, if a user drops a diagram, the agent can state, "I see the flowchart you provided," and answer questions about its content.

## Implementation Examples

### Basic Hook Usage

You can leverage the drag-and-drop functionality in custom components by importing the hook directly:

```typescript
import useDragDrop from '@/hooks/useDragDrop';

export default function ImageDropZone() {
  const { isDragging, handleDrop } = useDragDrop({
    onImage: (dataUrl: string, fileName: string) => {
      console.log(`Received ${fileName}, size: ${dataUrl.length} chars`);
      // Forward to your agent service here
    },
  });

  return (
    <div 
      onDrop={handleDrop} 
      className={isDragging ? 'drag-active' : 'drag-inactive'}
    >
      {isDragging ? 'Release to upload image' : 'Drag image here'}
    </div>
  );
}

```

### ChatWindow Integration

The standard implementation wraps the chat input area with the hook's handlers:

```tsx
// Simplified excerpt from components/ChatWindow.tsx
const ChatWindow = () => {
  const { isDragging, handleDrop } = useDragDrop({
    onImage: (dataUrl, fileName) => {
      const toolCall = {
        type: 'toolCall',
        name: 'image',
        arguments: { dataUrl, fileName }
      };
      appendToConversation(toolCall);
    },
  });

  return (
    <div onDrop={handleDrop} className={isDragging ? 'overlay' : ''}>
      <MessageList />
      <InputArea />
    </div>
  );
};

```

### Generated Payload Structure

When an image is successfully processed, the system generates the following JSON structure for the agent:

```json
{
  "type": "toolCall",
  "id": "img_1703123456789",
  "name": "image",
  "arguments": {
    "dataUrl": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ...",
    "fileName": "architecture_diagram.jpg"
  }
}

```

## Summary

- **`useDragDrop`** in [`hooks/useDragDrop.ts`](https://github.com/agegr/pi-web/blob/main/hooks/useDragDrop.ts) handles all browser drag events and converts image files to base-64 data URLs using the FileReader API.
- **`ChatWindow`** in [`components/ChatWindow.tsx`](https://github.com/agegr/pi-web/blob/main/components/ChatWindow.tsx) transforms these data URLs into structured tool call payloads with `type: "toolCall"` and `name: "image"`.
- The agent receives visual content as part of its prompt context through the tool call mechanism, enabling image-aware responses without modifying core chat infrastructure.
- This architecture maintains separation between UI interaction logic and agent communication protocols while supporting real-time visual analysis.

## Frequently Asked Questions

### How does Pi Web convert dropped images for the agent?

Pi Web uses the **`FileReader`** API to read the binary data of dropped files, converting them to base-64 encoded data URLs via `readAsDataURL()`. These strings are then packaged into tool call objects that the agent processes as prompt context.

### What file types are supported for drag-and-drop?

The **`useDragDrop`** hook filters files by MIME type, accepting standard image formats including `image/png`, `image/jpeg`, `image/gif`, and `image/webp`. Non-image files are ignored during the drop event and never reach the conversion stage.

### Can I trigger image tool calls programmatically without dragging?

Yes. While the hook handles drag-and-drop events, you can manually construct the same payload structure and call the `onImage` callback—or directly append a tool call message to the conversation stream—bypassing the file input handlers entirely.

### Where is the image data stored before being sent to the agent?

Image data exists temporarily as base-64 strings in the React component state managed by **`ChatWindow`**. The data is transmitted to the backend as part of the tool call payload and is not persisted to local storage or disk unless explicitly implemented by additional application logic.