# Designing Multimodal AI Agents with Vision Capabilities: A Complete Implementation Guide

> Learn to design multimodal AI agents with vision capabilities. Explore native vision, extract-to-text, and tool-on-demand modes using the MultimodalAgent class.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-08-01

---

**Multimodal AI agents with vision capabilities can process images, audio, and PDFs through three extraction modes—native vision, extract-to-text, and tool-on-demand—using the `MultimodalAgent` class in the bojieli/ai-agent-book repository.**

Building AI agents that truly understand visual content requires more than slapping an image API onto a chatbot. The **AI-Agent-Book** repository provides a production-ready framework for designing multimodal AI agents with vision capabilities, with its core implementation living in `chapter3/multimodal-agent/`. This guide walks you through the architecture, extraction modes, and hands-on code examples to build agents that see, listen, and read.

## Architecture of the Multimodal Agent

The vision-enabled system centers on three interconnected components that handle everything from media encoding to provider-agnostic API calls.

### MultimodalAgent: The Orchestration Layer

The **`MultimodalAgent`** class in [`chapter3/multimodal-agent/agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/agent.py) serves as the central hub. It manages conversation history, provider selection, and streaming responses while supporting three distinct vision processing strategies.

Key capabilities include:

- **Three extraction modes**: `NATIVE` (model-native vision), `EXTRACT_TO_TEXT` (OCR/transcription first), and `TOOL_ON_DEMAND` (function-calling)
- **Multi-provider support**: Gemini, OpenAI (gpt-5.6-luna), Doubao, with automatic OpenRouter fallback
- **Streaming with thinking output**: Gemini's chain-of-thought is printed to `stderr` while final answers stream to the caller

The constructor initializes these paths at lines 7-14, while the `chat` method (lines 844-864) dispatches to provider-specific handlers like `_process_native_gemini` (lines 375-422).

### MultimodalContent: Media Abstraction

Every image, audio clip, or PDF flows through the **`MultimodalContent`** dataclass (lines 51-73 in [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py)). This abstraction:

- Auto-detects MIME types from file extensions
- Provides `get_bytes()` for Gemini's raw-byte API and `get_base64()` for OpenAI-compatible endpoints
- Handles both local paths and URLs uniformly

```python
from chapter3.multimodal_agent.agent import MultimodalContent

# Load from filesystem

image = MultimodalContent(type="image", path="demo/cat.jpg")

# Or from URL

remote_image = MultimodalContent(
    type="image",
    path="https://example.com/chart.png",
    mime_type="image/png"  # optional override

)

```

### MultimodalTools: Explicit Vision Functions

When **`enable_multimodal_tools=True`**, the agent exposes `analyze_image`, `analyze_audio`, and `analyze_pdf` as callable functions. The model can request these via OpenAI-style function calling, the agent executes them through `_execute_tool` (lines 711-798), and injects results back into the conversation.

### Configuration and Provider Management

The **`Config`** class in [`chapter3/multimodal-agent/config.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/config.py) centralizes provider logic:

- **`_openrouter_model_id`**: Maps native model names to OpenRouter identifiers
- **`use_openrouter`**: Decides when to route through OpenRouter based on API key availability
- **`openai_client_args`**: Returns proper kwargs for streaming vs. non-streaming calls

The **`ExtractionMode`** enum (lines 37-41) defines the three processing strategies that fundamentally change how vision capabilities are leveraged.

## Three Extraction Modes for Vision Processing

| Mode | Use Case | Implementation |
|------|----------|----------------|
| **NATIVE** | Direct vision understanding with Gemini, GPT-5.6-luna, or Doubao | Media passed directly to model's vision endpoint |
| **EXTRACT_TO_TEXT** | OCR documents, transcribe audio for text-heavy processing | `load_and_extract_content()` runs extraction first |
| **TOOL_ON_DEMAND** | Explicit control over when vision occurs | Function definitions registered, model calls tools explicitly |

### Native Vision Mode

Native mode delivers the fastest inference by sending media directly to vision-capable models. The implementation branches by provider:

**Gemini path**: Uses `client.models.generate_content` with `types.Part.from_bytes` in `_process_native_gemini` and `_stream_gemini_response`.

**OpenAI path**: Uses `client.chat.completions.create` with base64-encoded `image_url` objects in `_process_native_openai` and `_analyze_with_openai`.

**Doubao path**: OpenAI-compatible endpoint via `AsyncOpenAI` with custom base URL in `_process_native_doubao`.

### Extract-to-Text Mode

For documents requiring precise text extraction or when using non-vision models, **`ExtractionMode.EXTRACT_TO_TEXT`** preprocesses media into plain text. The `load_and_extract_content` method (lines 1128-1130) handles this transformation before adding content to conversation history.

### Tool-on-Demand Mode

The most flexible approach registers vision capabilities as explicit tools. When `enable_multimodal_tools=True`, the agent:

1. Registers function definitions via `set_multimodal_tools_enabled` (lines 330-405)
2. Waits for the model to request analysis
3. Executes the appropriate tool and returns structured results

This gives the model agency to decide *whether* and *when* to analyze visual content.

## Implementing Vision-Capable Agents: Code Examples

### Basic Native Vision with Gemini

Start with the simplest case—direct image understanding without tool overhead:

```python
from chapter3.multimodal_agent.agent import MultimodalAgent, MultimodalContent

# Initialize with default settings (native mode, tools disabled)

agent = MultimodalAgent(enable_tools=False)

# Load image and prepare query

image = MultimodalContent(type="image", path="demo/cat.jpg")
query = "What is happening in this picture?"

# Stream the response

async for chunk in agent.chat(message=query, multimodal_content=image):
    print(chunk, end="")

```

The agent auto-selects the default model (Doubao-1.6) and routes to Gemini if configured, as implemented in `chat` (lines 844-864) and `_process_native_gemini` (lines 375-422).

### Enabling Function-Calling for Vision Tools

For scenarios requiring explicit tool control:

```python
from chapter3.multimodal_agent.agent import MultimodalAgent

# Enable multimodal tools—agent exposes analyze_image, analyze_audio, analyze_pdf

agent = MultimodalAgent(enable_tools=True)

# Prompt that encourages tool usage

prompt = "Identify the objects in the picture at /tmp/dog.png and tell me their colors."

# Model will request analyze_image tool automatically

async for chunk in agent.chat(prompt):
    print(chunk, end="")

```

The tool registration happens in `set_multimodal_tools_enabled` (lines 330-405), with execution routed through `_execute_tool` (lines 711-798).

### Extract-to-Text for Document Processing

When working with PDFs or requiring text-first processing:

```python
from chapter3.multimodal_agent.agent import MultimodalAgent, MultimodalContent
from chapter3.multimodal_agent.config import ExtractionMode

# Force extract-to-text mode

agent = MultimodalAgent(mode=ExtractionMode.EXTRACT_TO_TEXT)

# Load and extract PDF content

pdf = MultimodalContent(type="pdf", path="reports/annual.pdf")
await agent.load_and_extract_content(pdf)  # Adds extracted text to history

# Query the extracted content

async for chunk in agent.chat("Summarize the main findings of this report."):
    print(chunk, end="")

```

The `ExtractionMode` enum is defined at [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py) lines 37-41, with `load_and_extract_content` at lines 1128-1130.

## Provider-Specific Implementation Details

Each vision provider requires distinct handling in the source code:

| Provider | Key Method | Media Encoding |
|----------|-----------|----------------|
| Gemini | `_process_native_gemini` | Raw bytes via `types.Part.from_bytes` |
| OpenAI GPT-5.6-luna | `_process_native_openai` | Base64 data URIs in `image_url` |
| Doubao | `_process_native_doubao` | OpenAI-compatible base64 encoding |
| OpenRouter (fallback) | `openai_client_args` selector | OpenAI-compatible, model ID remapped |

The OpenRouter fallback activates automatically when primary provider keys are missing, with `_openrouter_model_id` handling the name translation.

## Streaming and Chain-of-Thought Handling

For Gemini specifically, the implementation preserves thinking output:

```python

# Pseudocode from _stream_gemini_response in agent.py

async for chunk in client.models.generate_content_stream(..., thinking_config=...):
    if chunk.is_thought:
        print(chunk.text, file=sys.stderr)  # Thinking to stderr

    else:
        yield chunk.text  # Final answer to caller

```

OpenAI streaming uses the standard `chat.completions.create` async iterator in `_stream_openai_response`.

## Key Files and Entry Points

| File | Purpose | Direct Link |
|------|---------|-------------|
| [`agent.py`](https://github.com/bojieli/ai-agent-book/blob/main/agent.py) | Core implementation: `MultimodalAgent`, `MultimodalTools`, `MultimodalContent` | [View source](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/agent.py) |
| [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py) | `Config`, `ModelConfig`, `ExtractionMode`, OpenRouter logic | [View source](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/config.py) |
| [`quickstart.py`](https://github.com/bojieli/ai-agent-book/blob/main/quickstart.py) | Minimal runnable example | [View source](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/quickstart.py) |
| [`demo.py`](https://github.com/bojieli/ai-agent-book/blob/main/demo.py) | Interactive demo with mode/tool toggles | [View source](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/demo.py) |
| [`tests/test_multimodal.py`](https://github.com/bojieli/ai-agent-book/blob/main/tests/test_multimodal.py) | Unit tests for native vision and tool flags | [View source](https://github.com/bojieli/ai-agent-book/blob/main/chapter3/multimodal-agent/test_multimodal.py) |

## Summary

- **Three extraction modes** (`NATIVE`, `EXTRACT_TO_TEXT`, `TOOL_ON_DEMAND`) provide flexibility in how vision capabilities are applied
- **`MultimodalAgent`** orchestrates provider-agnostic vision processing with automatic fallback to OpenRouter
- **`MultimodalContent`** abstracts media handling, auto-detecting MIME types and providing encoder helpers
- Function-calling tools give models explicit control over when to invoke vision analysis
- Gemini, OpenAI, and Doubao implementations use provider-specific endpoints while exposing a unified interface

## Frequently Asked Questions

### What is the difference between native vision mode and tool-on-demand mode?

Native mode passes images directly to vision-capable models for immediate processing, while tool-on-demand mode registers vision functions that the model must explicitly request. Native mode is faster and simpler; tool mode gives you audit trails and fine-grained control over which analyses run.

### How does the OpenRouter fallback work?

When a primary provider API key is missing, `use_openrouter` in [`config.py`](https://github.com/bojieli/ai-agent-book/blob/main/config.py) activates and `openai_client_args` returns OpenRouter endpoints with the model name remapped by `_openrouter_model_id`. This happens transparently without code changes.

### Can I process audio and PDFs with the same interface?

Yes. The `MultimodalContent` dataclass handles `type="audio"` and `type="pdf"` identically to images, with `MultimodalTools.analyze_audio` and `analyze_pdf` providing provider-specific implementations. Extract-to-text mode works particularly well for PDFs requiring OCR.

### Where is the conversation history stored?

The `MultimodalAgent` maintains `self.messages` as a list of OpenAI-compatible message dictionaries. Content from `chat()` calls and tool results are automatically appended, with multimodal content properly formatted per provider requirements.