Designing Multimodal AI Agents with Vision Capabilities: A Complete Implementation Guide

Multimodal AI agents with vision capabilities can process images, audio, and PDFs through three extraction modes—native vision, extract-to-text, and tool-on-demand—using the MultimodalAgent class in the bojieli/ai-agent-book repository.

Building AI agents that truly understand visual content requires more than slapping an image API onto a chatbot. The AI-Agent-Book repository provides a production-ready framework for designing multimodal AI agents with vision capabilities, with its core implementation living in chapter3/multimodal-agent/. This guide walks you through the architecture, extraction modes, and hands-on code examples to build agents that see, listen, and read.

Architecture of the Multimodal Agent

The vision-enabled system centers on three interconnected components that handle everything from media encoding to provider-agnostic API calls.

MultimodalAgent: The Orchestration Layer

The MultimodalAgent class in chapter3/multimodal-agent/agent.py serves as the central hub. It manages conversation history, provider selection, and streaming responses while supporting three distinct vision processing strategies.

Key capabilities include:

• Three extraction modes: NATIVE (model-native vision), EXTRACT_TO_TEXT (OCR/transcription first), and TOOL_ON_DEMAND (function-calling)
• Multi-provider support: Gemini, OpenAI (gpt-5.6-luna), Doubao, with automatic OpenRouter fallback
• Streaming with thinking output: Gemini's chain-of-thought is printed to stderr while final answers stream to the caller

The constructor initializes these paths at lines 7-14, while the chat method (lines 844-864) dispatches to provider-specific handlers like _process_native_gemini (lines 375-422).

MultimodalContent: Media Abstraction

Every image, audio clip, or PDF flows through the MultimodalContent dataclass (lines 51-73 in agent.py). This abstraction:

• Auto-detects MIME types from file extensions
• Provides get_bytes() for Gemini's raw-byte API and get_base64() for OpenAI-compatible endpoints
• Handles both local paths and URLs uniformly

from chapter3.multimodal_agent.agent import MultimodalContent

# Load from filesystem

image = MultimodalContent(type="image", path="demo/cat.jpg")

# Or from URL

remote_image = MultimodalContent(
    type="image",
    path="https://example.com/chart.png",
    mime_type="image/png"  # optional override

)

MultimodalTools: Explicit Vision Functions

When enable_multimodal_tools=True, the agent exposes analyze_image, analyze_audio, and analyze_pdf as callable functions. The model can request these via OpenAI-style function calling, the agent executes them through _execute_tool (lines 711-798), and injects results back into the conversation.

Configuration and Provider Management

The Config class in chapter3/multimodal-agent/config.py centralizes provider logic:

• _openrouter_model_id: Maps native model names to OpenRouter identifiers
• use_openrouter: Decides when to route through OpenRouter based on API key availability
• openai_client_args: Returns proper kwargs for streaming vs. non-streaming calls

The ExtractionMode enum (lines 37-41) defines the three processing strategies that fundamentally change how vision capabilities are leveraged.

Three Extraction Modes for Vision Processing

Mode	Use Case	Implementation
NATIVE	Direct vision understanding with Gemini, GPT-5.6-luna, or Doubao	Media passed directly to model's vision endpoint
EXTRACT_TO_TEXT	OCR documents, transcribe audio for text-heavy processing	load_and_extract_content() runs extraction first
TOOL_ON_DEMAND	Explicit control over when vision occurs	Function definitions registered, model calls tools explicitly

Native Vision Mode

Native mode delivers the fastest inference by sending media directly to vision-capable models. The implementation branches by provider:

Gemini path: Uses client.models.generate_content with types.Part.from_bytes in _process_native_gemini and _stream_gemini_response.

OpenAI path: Uses client.chat.completions.create with base64-encoded image_url objects in _process_native_openai and _analyze_with_openai.

Doubao path: OpenAI-compatible endpoint via AsyncOpenAI with custom base URL in _process_native_doubao.

Extract-to-Text Mode

For documents requiring precise text extraction or when using non-vision models, ExtractionMode.EXTRACT_TO_TEXT preprocesses media into plain text. The load_and_extract_content method (lines 1128-1130) handles this transformation before adding content to conversation history.

Tool-on-Demand Mode

The most flexible approach registers vision capabilities as explicit tools. When enable_multimodal_tools=True, the agent:

1. Registers function definitions via set_multimodal_tools_enabled (lines 330-405)
2. Waits for the model to request analysis
3. Executes the appropriate tool and returns structured results

This gives the model agency to decide whether and when to analyze visual content.

Implementing Vision-Capable Agents: Code Examples

Basic Native Vision with Gemini

Start with the simplest case—direct image understanding without tool overhead:

from chapter3.multimodal_agent.agent import MultimodalAgent, MultimodalContent

# Initialize with default settings (native mode, tools disabled)

agent = MultimodalAgent(enable_tools=False)

# Load image and prepare query

image = MultimodalContent(type="image", path="demo/cat.jpg")
query = "What is happening in this picture?"

# Stream the response

async for chunk in agent.chat(message=query, multimodal_content=image):
    print(chunk, end="")

The agent auto-selects the default model (Doubao-1.6) and routes to Gemini if configured, as implemented in chat (lines 844-864) and _process_native_gemini (lines 375-422).

Enabling Function-Calling for Vision Tools

For scenarios requiring explicit tool control:

from chapter3.multimodal_agent.agent import MultimodalAgent

# Enable multimodal tools—agent exposes analyze_image, analyze_audio, analyze_pdf

agent = MultimodalAgent(enable_tools=True)

# Prompt that encourages tool usage

prompt = "Identify the objects in the picture at /tmp/dog.png and tell me their colors."

# Model will request analyze_image tool automatically

async for chunk in agent.chat(prompt):
    print(chunk, end="")

The tool registration happens in set_multimodal_tools_enabled (lines 330-405), with execution routed through _execute_tool (lines 711-798).

Extract-to-Text for Document Processing

When working with PDFs or requiring text-first processing:

from chapter3.multimodal_agent.agent import MultimodalAgent, MultimodalContent
from chapter3.multimodal_agent.config import ExtractionMode

# Force extract-to-text mode

agent = MultimodalAgent(mode=ExtractionMode.EXTRACT_TO_TEXT)

# Load and extract PDF content

pdf = MultimodalContent(type="pdf", path="reports/annual.pdf")
await agent.load_and_extract_content(pdf)  # Adds extracted text to history

# Query the extracted content

async for chunk in agent.chat("Summarize the main findings of this report."):
    print(chunk, end="")

The ExtractionMode enum is defined at config.py lines 37-41, with load_and_extract_content at lines 1128-1130.

Provider-Specific Implementation Details

Each vision provider requires distinct handling in the source code:

Provider	Key Method	Media Encoding
Gemini	_process_native_gemini	Raw bytes via types.Part.from_bytes
OpenAI GPT-5.6-luna	_process_native_openai	Base64 data URIs in image_url
Doubao	_process_native_doubao	OpenAI-compatible base64 encoding
OpenRouter (fallback)	openai_client_args selector	OpenAI-compatible, model ID remapped

The OpenRouter fallback activates automatically when primary provider keys are missing, with _openrouter_model_id handling the name translation.

Streaming and Chain-of-Thought Handling

For Gemini specifically, the implementation preserves thinking output:

# Pseudocode from _stream_gemini_response in agent.py

async for chunk in client.models.generate_content_stream(..., thinking_config=...):
    if chunk.is_thought:
        print(chunk.text, file=sys.stderr)  # Thinking to stderr

    else:
        yield chunk.text  # Final answer to caller

OpenAI streaming uses the standard chat.completions.create async iterator in _stream_openai_response.

Key Files and Entry Points

File	Purpose	Direct Link
agent.py	Core implementation: MultimodalAgent, MultimodalTools, MultimodalContent	View source
config.py	Config, ModelConfig, ExtractionMode, OpenRouter logic	View source
quickstart.py	Minimal runnable example	View source
demo.py	Interactive demo with mode/tool toggles	View source
tests/test_multimodal.py	Unit tests for native vision and tool flags	View source

Summary

• Three extraction modes (NATIVE, EXTRACT_TO_TEXT, TOOL_ON_DEMAND) provide flexibility in how vision capabilities are applied
• MultimodalAgent orchestrates provider-agnostic vision processing with automatic fallback to OpenRouter
• MultimodalContent abstracts media handling, auto-detecting MIME types and providing encoder helpers
• Function-calling tools give models explicit control over when to invoke vision analysis
• Gemini, OpenAI, and Doubao implementations use provider-specific endpoints while exposing a unified interface

Frequently Asked Questions

What is the difference between native vision mode and tool-on-demand mode?

Native mode passes images directly to vision-capable models for immediate processing, while tool-on-demand mode registers vision functions that the model must explicitly request. Native mode is faster and simpler; tool mode gives you audit trails and fine-grained control over which analyses run.

How does the OpenRouter fallback work?

When a primary provider API key is missing, use_openrouter in config.py activates and openai_client_args returns OpenRouter endpoints with the model name remapped by _openrouter_model_id. This happens transparently without code changes.

Can I process audio and PDFs with the same interface?

Yes. The MultimodalContent dataclass handles type="audio" and type="pdf" identically to images, with MultimodalTools.analyze_audio and analyze_pdf providing provider-specific implementations. Extract-to-text mode works particularly well for PDFs requiring OCR.

Where is the conversation history stored?

The MultimodalAgent maintains self.messages as a list of OpenAI-compatible message dictionaries. Content from chat() calls and tool results are automatically appended, with multimodal content properly formatted per provider requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}