# How to Implement Multimodal Agents with Vision Capabilities: A Complete Guide to AI-Agent-Book

> Implement multimodal agents with vision capabilities by setting use_vision=True in the Agent class. This guide shows how to inject screenshots for advanced AI interactions.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-26

---

**You can implement multimodal agents with vision capabilities by setting `use_vision=True` when initializing the `Agent` class, which automatically injects screenshots into the message stream using the `ImageURL` and `ContentPartImageParam` models defined in the framework's message schema.**

The `bojieli/ai-agent-book` repository provides a production-ready framework for building agents that process both text and visual inputs. Implementing multimodal agents with vision capabilities requires understanding how the framework standardizes image data within OpenAI-compatible message schemas and orchestrates screenshot capture through Playwright integration. This architecture separates message representation, agent logic, and LLM provider adaptation into distinct layers.

## Architecture of Vision-Enabled Agents

The repository implements vision support through a modular architecture that wraps image data in standardized message objects and controls vision capabilities via configuration flags.

### Message Schema and Image Representation

The foundation of multimodal support lies in [`browser_use/llm/messages.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/messages.py). The `ImageURL` class (lines 54-71) encapsulates image data with three key attributes:

- **`url`**: Supports both external URLs and base64-encoded data URLs
- **`detail`**: Controls processing fidelity with values `"auto"`, `"low"`, or `"high"`
- **`media_type`**: Specifies the image format (e.g., `"image/png"`)

These image objects integrate into mixed-content messages through `ContentPartImageParam` and `ContentPartTextParam` (lines 75-84). The `UserMessage` model accepts a `content` list containing both text and image parts, allowing a single message to carry multimodal context. The `UserMessage.text` property provides convenient access to textual portions while preserving the full `content` list for LLM consumption.

### Agent Configuration and Vision Toggling

Vision capabilities are controlled through the `Agent` class in [`browser_use/agent/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/agent/service.py) (lines 156-178). The constructor accepts a boolean `use_vision` parameter that determines whether the agent injects visual context into messages.

When `use_vision=True`, the agent automatically:

1. Triggers screenshot capture via the screenshot service
2. Wraps image data in `ImageURL` objects
3. Appends `ContentPartImageParam` instances to the `UserMessage.content` list
4. Passes the multimodal payload to the configured LLM backend

You can fine-tune vision costs by setting `vision_detail_level` to `"low"` for reduced token consumption or `"high"` for maximum visual fidelity.

### Screenshot Capture and LLM Adaptation

The [`browser_use/screenshots/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/screenshots/service.py) module handles visual data generation using Playwright. It captures page snapshots, stores them locally, and returns data URLs that feed directly into `ImageURL` objects.

LLM backend adapters in files like [`browser_use/llm/openai/chat.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/openai/chat.py) handle the provider-specific serialization. These adapters translate the generic `UserMessage.content` list—including any embedded `ImageURL` objects—into the required API format, such as OpenAI's `multipart/form-data` vision requests.

## Enabling Vision in Your Agent

Follow these steps to convert a text-only agent into a multimodal implementation:

1. **Initialize a vision-capable LLM backend** (OpenAI GPT-4 Vision or compatible)

2. **Set the vision flag** when constructing the agent:
   
   ```python
   agent = Agent(
       task="Extract pricing information from the product page",
       llm=llm,
       use_vision=True
   )
   ```

3. **(Optional) Configure detail level** to balance cost and accuracy:
   
   ```python
   agent.settings.vision_detail_level = "low"  # Options: "auto", "low", "high"

   ```

4. **Execute the task**—the framework automatically captures screenshots and injects them into the message stream before each LLM call.

## Code Examples

### Minimal Multimodal Agent Setup

This example demonstrates the complete setup for a vision-enabled agent using the OpenAI backend:

```python
from browser_use.agent.service import Agent
from browser_use.llm.openai.chat import OpenAIChat

# Initialize a vision-capable LLM

llm = OpenAIChat(model="gpt-4-vision-preview", api_key="YOUR_KEY")

# Create agent with vision enabled

agent = Agent(
    task="Summarize the key visual elements on the current page",
    llm=llm,
    use_vision=True,                # Enable vision capabilities

    vision_detail_level="auto",     # Control processing fidelity

)

# Run the agent - screenshots are captured automatically

result = agent.run()
print(result)

```

*Key implementation details:*
- `Agent` constructor: [`browser_use/agent/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/agent/service.py) (lines 156-178)
- OpenAI vision backend: [`browser_use/llm/openai/chat.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/openai/chat.py)

### Manually Adding Images to Messages

For scenarios requiring custom images rather than browser screenshots, manually construct multimodal messages:

```python
from browser_use.llm.messages import UserMessage, ContentPartImageParam, ImageURL
import base64
import pathlib

def png_to_data_url(path):
    """Helper to encode local images as data URLs"""
    data = pathlib.Path(path).read_bytes()
    b64 = base64.b64encode(data).decode()
    return f"data:image/png;base64,{b64}"

# Create image reference with detail level

img_url = ImageURL(
    url=png_to_data_url("diagram.png"),
    detail="high",
    media_type="image/png",
)

# Build multimodal message

msg = UserMessage(
    content=[
        ContentPartImageParam(image_url=img_url),
        ContentPartTextParam(text="Analyze the workflow diagram shown above"),
    ]
)

# Send via LLM client

response = llm.chat([msg])

```

*Implementation references:*
- `ImageURL` model: [`browser_use/llm/messages.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/messages.py) (lines 54-65)
- Content parts: [`browser_use/llm/messages.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/messages.py) (lines 75-84)

## Key Implementation Files

Understanding these source files is essential for customizing vision capabilities:

- **[`browser_use/llm/messages.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/messages.py)** — Defines `ImageURL`, `ContentPartImageParam`, and `UserMessage` schemas for multimodal content
- **[`browser_use/agent/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/agent/service.py)** — Contains the `Agent` class with `use_vision` flag and message assembly logic (lines 156-178)
- **[`browser_use/screenshots/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/screenshots/service.py)** — Playwright-based screenshot capture service that generates image data for vision inputs
- **[`browser_use/llm/openai/chat.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/llm/openai/chat.py)** — OpenAI-specific backend that serializes mixed text/image content into API requests
- **[`browser_use/examples/use-cases/shopping.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/examples/use-cases/shopping.py)** — Working example of vision-enabled e-commerce automation

## Summary

- **Toggle vision capabilities** by setting `use_vision=True` in the `Agent` constructor defined in [`browser_use/agent/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/agent/service.py)
- **Control processing costs** using the `detail` parameter on `ImageURL` objects with levels `"auto"`, `"low"`, or `"high"`
- **Construct multimodal messages** by combining `ContentPartImageParam` and `ContentPartTextParam` within a `UserMessage` object
- **Capture visual context automatically** via the screenshot service in [`browser_use/screenshots/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/screenshots/service.py), or manually inject base64-encoded images
- **Support multiple LLM providers** through backend adapters that translate the generic message schema into provider-specific vision API formats

## Frequently Asked Questions

### What does the `use_vision` flag control in the AI-Agent-Book framework?

The `use_vision` boolean parameter in the `Agent` class (located in [`browser_use/agent/service.py`](https://github.com/bojieli/ai-agent-book/blob/main/browser_use/agent/service.py)) determines whether the agent automatically captures browser screenshots and injects them into the LLM message stream. When enabled, the agent wraps screenshot data in `ImageURL` objects and appends them to `UserMessage.content` lists before each inference call, allowing the model to "see" the current page state.

### How do detail levels affect vision processing costs?

The `detail` field on `ImageURL` objects accepts three values: `"low"` uses fewer tokens for faster, cheaper processing, `"high"` provides maximum visual fidelity at higher token cost, and `"auto"` lets the LLM provider choose based on image size. This setting directly impacts API costs and response latency, making it essential for production deployments where you must balance accuracy against budget constraints.

### Can I implement vision capabilities with LLM providers other than OpenAI?

Yes. While the examples reference OpenAI's GPT-4 Vision, the framework's architecture supports multiple providers through adapter classes in `browser_use/llm/`. Each provider implementation (Gemini, Anthropic, etc.) handles the serialization of `UserMessage.content` containing `ContentPartImageParam` objects into the provider's specific multimodal API format, abstracting provider differences from the agent logic.

### How do I add custom images instead of browser screenshots?

You can manually construct multimodal messages by creating `ImageURL` objects with base64-encoded data URLs and wrapping them in `ContentPartImageParam` instances. Add these to a `UserMessage.content` list alongside `ContentPartTextParam` objects, then pass the message directly to your LLM backend's `chat()` method, bypassing the automatic screenshot service entirely.