How Pi Web Handles Image Drag-and-Drop with Agent Prompt Integration

Pi Web converts dropped images into base-64 data URLs and injects them as structured tool calls into the agent's prompt context, allowing the LLM to analyze visual content in real time.

The agegr/pi-web repository implements a seamless image drag-and-drop system that bridges browser file handling with LLM agent capabilities. By leveraging the useDragDrop hook and the ChatWindow component, the application transforms raw image files into contextual prompt data that the agent can reference and reason about during conversations.

The Drag-and-Drop Architecture

Event Handling with useDragDrop

The core functionality resides in hooks/useDragDrop.ts, which registers global browser listeners for drag-enter, drag-over, and drop events. When a user drags an image file over the chat interface, the hook sets a reactive state flag that provides immediate visual feedback. On drop, it intercepts the native DragEvent and accesses event.dataTransfer.files to retrieve the file list.

The hook filters incoming files by MIME type to ensure only image formats (such as image/png, image/jpeg, or image/webp) are processed. This validation occurs client-side before any data conversion begins, preventing invalid file types from reaching the agent pipeline.

File Processing and Base-64 Conversion

Once validated, the hook instantiates a FileReader to asynchronously read the binary image data. It converts the file to a base-64 encoded data URL using readAsDataURL(), producing a string in the format data:image/*;base64,…. This string representation is necessary because the agent prompt system requires text-serializable content.

The hook then invokes a callback—typically provided by the parent component—with two arguments: the base-64 data URL and the original filename. This decouples the file handling logic from the chat state management, following the single-responsibility principle.

Agent Prompt Integration via Tool Calls

Tool Call Construction in ChatWindow

The components/ChatWindow.tsx file orchestrates the integration between the UI and the agent. When useDragDrop returns an image payload, ChatWindow constructs a specialized tool call object that conforms to Pi Web's internal messaging protocol. The payload structure follows this schema:

{
  "type": "toolCall",
  "id": "t1",
  "name": "image",
  "arguments": {
    "dataUrl": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAA...",
    "fileName": "snapshot.png"
  }
}

ChatWindow appends this object to the conversation history as a new message entry. By treating the image as a tool call rather than plain text, the system maintains type safety and allows the backend to apply specific preprocessing logic for visual inputs before they reach the LLM.

How the Agent Processes Visual Input

When the conversation stream reaches the agent, the backend detects the image tool call and extracts the base-64 data. The agent framework converts this data into a format the underlying LLM can interpret—typically by inserting a descriptive marker or processing the image through a vision-capable model endpoint.

The agent then receives the image as part of its prompt context, enabling it to reference specific visual elements in its response. For example, if a user drops a diagram, the agent can state, "I see the flowchart you provided," and answer questions about its content.

Implementation Examples

Basic Hook Usage

You can leverage the drag-and-drop functionality in custom components by importing the hook directly:

import useDragDrop from '@/hooks/useDragDrop';

export default function ImageDropZone() {
  const { isDragging, handleDrop } = useDragDrop({
    onImage: (dataUrl: string, fileName: string) => {
      console.log(`Received ${fileName}, size: ${dataUrl.length} chars`);
      // Forward to your agent service here
    },
  });

  return (
    <div 
      onDrop={handleDrop} 
      className={isDragging ? 'drag-active' : 'drag-inactive'}
    >
      {isDragging ? 'Release to upload image' : 'Drag image here'}
    </div>
  );
}

ChatWindow Integration

The standard implementation wraps the chat input area with the hook's handlers:

// Simplified excerpt from components/ChatWindow.tsx
const ChatWindow = () => {
  const { isDragging, handleDrop } = useDragDrop({
    onImage: (dataUrl, fileName) => {
      const toolCall = {
        type: 'toolCall',
        name: 'image',
        arguments: { dataUrl, fileName }
      };
      appendToConversation(toolCall);
    },
  });

  return (
    <div onDrop={handleDrop} className={isDragging ? 'overlay' : ''}>
      <MessageList />
      <InputArea />
    </div>
  );
};

Generated Payload Structure

When an image is successfully processed, the system generates the following JSON structure for the agent:

{
  "type": "toolCall",
  "id": "img_1703123456789",
  "name": "image",
  "arguments": {
    "dataUrl": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQ...",
    "fileName": "architecture_diagram.jpg"
  }
}

Summary

• useDragDrop in hooks/useDragDrop.ts handles all browser drag events and converts image files to base-64 data URLs using the FileReader API.
• ChatWindow in components/ChatWindow.tsx transforms these data URLs into structured tool call payloads with type: "toolCall" and name: "image".
• The agent receives visual content as part of its prompt context through the tool call mechanism, enabling image-aware responses without modifying core chat infrastructure.
• This architecture maintains separation between UI interaction logic and agent communication protocols while supporting real-time visual analysis.

Frequently Asked Questions

How does Pi Web convert dropped images for the agent?

Pi Web uses the FileReader API to read the binary data of dropped files, converting them to base-64 encoded data URLs via readAsDataURL(). These strings are then packaged into tool call objects that the agent processes as prompt context.

What file types are supported for drag-and-drop?

The useDragDrop hook filters files by MIME type, accepting standard image formats including image/png, image/jpeg, image/gif, and image/webp. Non-image files are ignored during the drop event and never reach the conversion stage.

Can I trigger image tool calls programmatically without dragging?

Yes. While the hook handles drag-and-drop events, you can manually construct the same payload structure and call the onImage callback—or directly append a tool call message to the conversation stream—bypassing the file input handlers entirely.

Where is the image data stored before being sent to the agent?

Image data exists temporarily as base-64 strings in the React component state managed by ChatWindow. The data is transmitted to the backend as part of the tool call payload and is not persisted to local storage or disk unless explicitly implemented by additional application logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}