How to Implement Multimodal Agents with Vision Capabilities: A Complete Guide to AI-Agent-Book

You can implement multimodal agents with vision capabilities by setting use_vision=True when initializing the Agent class, which automatically injects screenshots into the message stream using the ImageURL and ContentPartImageParam models defined in the framework's message schema.

The bojieli/ai-agent-book repository provides a production-ready framework for building agents that process both text and visual inputs. Implementing multimodal agents with vision capabilities requires understanding how the framework standardizes image data within OpenAI-compatible message schemas and orchestrates screenshot capture through Playwright integration. This architecture separates message representation, agent logic, and LLM provider adaptation into distinct layers.

Architecture of Vision-Enabled Agents

The repository implements vision support through a modular architecture that wraps image data in standardized message objects and controls vision capabilities via configuration flags.

Message Schema and Image Representation

The foundation of multimodal support lies in browser_use/llm/messages.py. The ImageURL class (lines 54-71) encapsulates image data with three key attributes:

• url: Supports both external URLs and base64-encoded data URLs
• detail: Controls processing fidelity with values "auto", "low", or "high"
• media_type: Specifies the image format (e.g., "image/png")

These image objects integrate into mixed-content messages through ContentPartImageParam and ContentPartTextParam (lines 75-84). The UserMessage model accepts a content list containing both text and image parts, allowing a single message to carry multimodal context. The UserMessage.text property provides convenient access to textual portions while preserving the full content list for LLM consumption.

Agent Configuration and Vision Toggling

Vision capabilities are controlled through the Agent class in browser_use/agent/service.py (lines 156-178). The constructor accepts a boolean use_vision parameter that determines whether the agent injects visual context into messages.

When use_vision=True, the agent automatically:

1. Triggers screenshot capture via the screenshot service
2. Wraps image data in ImageURL objects
3. Appends ContentPartImageParam instances to the UserMessage.content list
4. Passes the multimodal payload to the configured LLM backend

You can fine-tune vision costs by setting vision_detail_level to "low" for reduced token consumption or "high" for maximum visual fidelity.

Screenshot Capture and LLM Adaptation

The browser_use/screenshots/service.py module handles visual data generation using Playwright. It captures page snapshots, stores them locally, and returns data URLs that feed directly into ImageURL objects.

LLM backend adapters in files like browser_use/llm/openai/chat.py handle the provider-specific serialization. These adapters translate the generic UserMessage.content list—including any embedded ImageURL objects—into the required API format, such as OpenAI's multipart/form-data vision requests.

Enabling Vision in Your Agent

Follow these steps to convert a text-only agent into a multimodal implementation:

1. Initialize a vision-capable LLM backend (OpenAI GPT-4 Vision or compatible)

2. Set the vision flag when constructing the agent:

   agent = Agent(
       task="Extract pricing information from the product page",
       llm=llm,
       use_vision=True
   )

3. (Optional) Configure detail level to balance cost and accuracy:

   agent.settings.vision_detail_level = "low"  # Options: "auto", "low", "high"
   

4. Execute the task—the framework automatically captures screenshots and injects them into the message stream before each LLM call.

Code Examples

Minimal Multimodal Agent Setup

This example demonstrates the complete setup for a vision-enabled agent using the OpenAI backend:

from browser_use.agent.service import Agent
from browser_use.llm.openai.chat import OpenAIChat

# Initialize a vision-capable LLM

llm = OpenAIChat(model="gpt-4-vision-preview", api_key="YOUR_KEY")

# Create agent with vision enabled

agent = Agent(
    task="Summarize the key visual elements on the current page",
    llm=llm,
    use_vision=True,                # Enable vision capabilities

    vision_detail_level="auto",     # Control processing fidelity

)

# Run the agent - screenshots are captured automatically

result = agent.run()
print(result)

Key implementation details:

• Agent constructor: browser_use/agent/service.py (lines 156-178)
• OpenAI vision backend: browser_use/llm/openai/chat.py

Manually Adding Images to Messages

For scenarios requiring custom images rather than browser screenshots, manually construct multimodal messages:

from browser_use.llm.messages import UserMessage, ContentPartImageParam, ImageURL
import base64
import pathlib

def png_to_data_url(path):
    """Helper to encode local images as data URLs"""
    data = pathlib.Path(path).read_bytes()
    b64 = base64.b64encode(data).decode()
    return f"data:image/png;base64,{b64}"

# Create image reference with detail level

img_url = ImageURL(
    url=png_to_data_url("diagram.png"),
    detail="high",
    media_type="image/png",
)

# Build multimodal message

msg = UserMessage(
    content=[
        ContentPartImageParam(image_url=img_url),
        ContentPartTextParam(text="Analyze the workflow diagram shown above"),
    ]
)

# Send via LLM client

response = llm.chat([msg])

Implementation references:

• ImageURL model: browser_use/llm/messages.py (lines 54-65)
• Content parts: browser_use/llm/messages.py (lines 75-84)

Key Implementation Files

Understanding these source files is essential for customizing vision capabilities:

• browser_use/llm/messages.py — Defines ImageURL, ContentPartImageParam, and UserMessage schemas for multimodal content
• browser_use/agent/service.py — Contains the Agent class with use_vision flag and message assembly logic (lines 156-178)
• browser_use/screenshots/service.py — Playwright-based screenshot capture service that generates image data for vision inputs
• browser_use/llm/openai/chat.py — OpenAI-specific backend that serializes mixed text/image content into API requests
• browser_use/examples/use-cases/shopping.py — Working example of vision-enabled e-commerce automation

Summary

• Toggle vision capabilities by setting use_vision=True in the Agent constructor defined in browser_use/agent/service.py
• Control processing costs using the detail parameter on ImageURL objects with levels "auto", "low", or "high"
• Construct multimodal messages by combining ContentPartImageParam and ContentPartTextParam within a UserMessage object
• Capture visual context automatically via the screenshot service in browser_use/screenshots/service.py, or manually inject base64-encoded images
• Support multiple LLM providers through backend adapters that translate the generic message schema into provider-specific vision API formats

Frequently Asked Questions

What does the use_vision flag control in the AI-Agent-Book framework?

The use_vision boolean parameter in the Agent class (located in browser_use/agent/service.py) determines whether the agent automatically captures browser screenshots and injects them into the LLM message stream. When enabled, the agent wraps screenshot data in ImageURL objects and appends them to UserMessage.content lists before each inference call, allowing the model to "see" the current page state.

How do detail levels affect vision processing costs?

The detail field on ImageURL objects accepts three values: "low" uses fewer tokens for faster, cheaper processing, "high" provides maximum visual fidelity at higher token cost, and "auto" lets the LLM provider choose based on image size. This setting directly impacts API costs and response latency, making it essential for production deployments where you must balance accuracy against budget constraints.

Can I implement vision capabilities with LLM providers other than OpenAI?

Yes. While the examples reference OpenAI's GPT-4 Vision, the framework's architecture supports multiple providers through adapter classes in browser_use/llm/. Each provider implementation (Gemini, Anthropic, etc.) handles the serialization of UserMessage.content containing ContentPartImageParam objects into the provider's specific multimodal API format, abstracting provider differences from the agent logic.

How do I add custom images instead of browser screenshots?

You can manually construct multimodal messages by creating ImageURL objects with base64-encoded data URLs and wrapping them in ContentPartImageParam instances. Add these to a UserMessage.content list alongside ContentPartTextParam objects, then pass the message directly to your LLM backend's chat() method, bypassing the automatic screenshot service entirely.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}