# How to Deploy AI Agents to Production: A Complete Guide to Model Serving, MCP Tools, and Orchestration

> Learn how to deploy AI agents to production. This guide covers model serving, MCP tools, and orchestration using Docker or Kubernetes for seamless AI agent deployment.

- Repository: [Bojie Li/ai-agent-book](https://github.com/bojieli/ai-agent-book)
- Tags: how-to-guide
- Published: 2026-07-30

---

**Deploying AI agents to production requires three layers: a model‑as‑a‑service backend (vLLM or Ollama), containerized MCP tool servers for execution and perception, and Docker or Kubernetes orchestration with proper secrets management.**

This guide walks you through the production deployment architecture defined in the bojieli/ai-agent-book repository, covering how to serve LLMs locally, run Model‑Context‑Protocol (MCP) tool containers, and orchestrate the entire stack using Docker Compose or Kubernetes.

## Architectural Blueprint for Production Deployment

Production deployment of an AI agent—which combines an LLM, context, and tools—depends on a three‑layer infrastructure: model serving, tool back‑ends, and orchestration.

### Model‑as‑a‑Service Layer

你 must expose the LLM through an OpenAI‑compatible API. According to the source code in `chapter2/local_llm_serving`, you have two primary options:

- **vLLM** – Use this CUDA‑enabled option on Linux GPUs for high‑throughput token generation with KV‑cache reuse.
- **Ollama** – Use this for macOS, Windows, or Linux without a GPU, pulling lightweight models like `qwen3:0.6b`.

The [`main.py`](https://github.com/bojieli/ai-agent-book/blob/main/main.py) wrapper auto‑detects the environment, launches the appropriate server ([`server.py`](https://github.com/bojieli/ai-agent-book/blob/main/server.py) for vLLM or [`ollama_native.py`](https://github.com/bojieli/ai-agent-book/blob/main/ollama_native.py) for Ollama), and exposes a unified endpoint at `POST /v1/chat/completions`. This design lets you swap backends without changing agent code, enabling seamless migration from local development (Ollama) to high‑performance GPU clusters (vLLM).

### MCP Tool Servers

The repository provides three categories of containerized MCP (Model‑Context‑Protocol) servers, each defined in `chapter4/`:

- **execution‑tools** – Sandboxed multi‑language code execution (Python, Node, Go, Java, C++, Rust, PHP, Bash). The Dockerfile installs language‑specific runtimes and mounts an `execution-workspace` volume for isolation.
- **perception‑tools** – Document ingestion, web search, OCR, and PDF parsing using Python stacks including `tesseract`, `requests`, and `BeautifulSoup`.
- **collaboration‑tools** – Browser automation, Excel processing, and human‑in‑the‑loop dialogs using Headless Chromium and `openpyxl`.

All services read a common `.env` file for API keys (OpenAI, Google, etc.) and expose the same MCP JSON contract, allowing the Agent to call any tool with a uniform request shape.

### Orchestration and Production Concerns

The [`chapter4/docker-compose.yml`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/docker-compose.yml) file defines the three MCP services with network isolation and shared volumes. For production environments, the repository recommends four critical practices:

1. **Secrets Management** – Replace the plain‑text `.env` file with Docker Swarm or Kubernetes secrets before deploying to production.
2. **Observability** – Each container logs to stdout; use `docker-compose logs -f` for streaming or attach Prometheus exporters for metrics.
3. **Scalability** – The [`chapter8/gaia-experience/AWorld/env/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/env/README.md) describes Kubernetes manifests for horizontal scaling of each MCP service and the model server independently.
4. **Separation of Concerns** – The Agent’s runtime never embeds model weights; it only communicates via HTTP to the model endpoint and tool services. This enables independent upgrades of models and tools, supporting continuous delivery.

## Step‑by‑Step Deployment Examples

### Launch the Full Stack with Docker Compose

To deploy AI agents to production using Docker, execute the following commands from the repository root:

```bash

# 1️⃣ Copy example env and fill in your keys

cp .env.example .env

# edit .env → set OPENAI_API_KEY, GOOGLE_API_KEY, …

# 2️⃣ Build and launch all services

docker-compose up -d

# 3️⃣ Verify health

curl -s http://localhost:8000/health

```

The [`docker-compose.yml`](https://github.com/bojieli/ai-agent-book/blob/main/docker-compose.yml) file orchestrates the execution‑tools, perception‑tools, and collaboration‑tools containers, pulling base images and building MCP containers automatically.

### Serve Local LLMs with vLLM or Ollama

For Linux GPU environments, start the vLLM server:

```bash
cd chapter2/local_llm_serving
pip install -r requirements.txt
python server.py  # starts vLLM on port 8000

```

Now instantiate the agent:

```python
from main import ToolCallingAgent

agent = ToolCallingAgent()  # auto‑detects the vLLM endpoint

response = agent.chat("Summarize the latest AI safety paper.")
print(response)

```

The `ToolCallingAgent` class handles endpoint detection and API compatibility, allowing the same code to work with either vLLM or Ollama backends.

### Invoke Tools from the Agent

To trigger tool calls (e.g., weather lookup), use:

```python
from main import ToolCallingAgent

agent = ToolCallingAgent()
for chunk in agent.chat("What is the current temperature in Tokyo?", stream=True):
    if chunk["type"] == "tool_result":
        print("🛠️ Tool output:", chunk["content"])

```

The request flow follows this path:
- Agent → **model** (`/v1/chat/completions`)
- LLM decides to call `get_current_temperature`
- MCP **perception‑tools** receives a JSON RPC call, fetches data from Open‑Meteo, and returns the result

The plumbing logic resides in [`chapter4/perception-tools/src/main.py`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/perception-tools/src/main.py).

### Scale with Kubernetes

For horizontal scaling, apply the Kubernetes manifests from `chapter8/gaia-experience/AWorld/env/`:

```bash
kubectl apply -f k8s/model-server.yaml
kubectl apply -f k8s/execution-tools.yaml
kubectl apply -f k8s/perception-tools.yaml
kubectl apply -f k8s/collaboration-tools.yaml

```

Each manifest references Docker images built from the Chapter 4 Dockerfiles. The [`README.md`](https://github.com/bojieli/ai-agent-book/blob/main/README.md) in `chapter8/gaia-experience/AWorld/env/` details the required secret objects and ingress configuration for production traffic.

## Critical Source Files and References

| File | Purpose |
|------|---------|
| [`chapter2/local_llm_serving/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter2/local_llm_serving/README.md) | Guide to running portable LLM servers and using the Python SDK. |
| [`chapter4/DOCKER_DEPLOYMENT.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/DOCKER_DEPLOYMENT.md) | Step‑by‑step Docker deployment instructions with security advice. |
| [`chapter4/docker-compose.yml`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/docker-compose.yml) | Single‑file composition of all three MCP services. |
| `chapter4/execution-tools/Dockerfile` | Multi‑language sandbox environment for code execution. |
| `chapter4/perception-tools/Dockerfile` | Document processing and web search capabilities. |
| `chapter4/collaboration-tools/Dockerfile` | Browser automation and Excel processing tools. |
| [`chapter8/gaia-experience/AWorld/env/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/env/README.md) | Production Kubernetes deployment and scaling strategies. |
| [`chapter8/gaia-experience/AWorld/aworlddistributed/README.md`](https://github.com/bojieli/ai-agent-book/blob/main/chapter8/gaia-experience/AWorld/aworlddistributed/README.md) | Distributed MCP server architecture for load‑balanced deployments. |

## Summary

- **Model Serving**: Use `chapter2/local_llm_serving` to expose vLLM or Ollama via an OpenAI‑compatible API, enabling backend swaps without code changes.
- **Tool Infrastructure**: Deploy execution, perception, and collaboration MCP servers using [`chapter4/docker-compose.yml`](https://github.com/bojieli/ai-agent-book/blob/main/chapter4/docker-compose.yml) or Kubernetes manifests from Chapter 8.
- **Security**: Replace `.env` files with proper secret management (Docker Swarm or Kubernetes secrets) before production deployment.
- **Scalability**: The decoupled architecture allows independent scaling of LLM inference and tool execution through container orchestration.
- **Integration**: The `ToolCallingAgent` class provides a unified Python interface that routes LLM calls and tool invocations to the appropriate back‑ends automatically.

## Frequently Asked Questions

### What is the difference between vLLM and Ollama for production deployment?

**vLLM** is optimized for CUDA‑enabled Linux servers and provides high‑throughput token generation with KV‑cache reuse, making it ideal for GPU clusters. **Ollama** runs on macOS, Windows, and Linux without requiring a GPU and pulls lightweight quantized models, making it suitable for development environments or edge deployment. The [`main.py`](https://github.com/bojieli/ai-agent-book/blob/main/main.py) wrapper in `chapter2/local_llm_serving` automatically detects which backend is available and exposes a uniform OpenAI‑compatible API.

### How do MCP tool servers handle security and isolation?

Each tool category runs in its own Docker container defined in separate Dockerfiles within `chapter4/`. The **execution‑tools** container uses a dedicated `execution-workspace` volume to sandbox multi‑language code execution (Python, Node, Go, Java, etc.), while **perception‑tools** and **collaboration‑tools** run isolated processes for document parsing and browser automation. All containers read secrets from environment variables injected via `.env` files or Kubernetes secret objects, ensuring API keys never reside in the application code.

### Can I scale individual components independently?

Yes. The architecture decouples the LLM inference layer from the tool execution layer, allowing you to scale each horizontally. Use the Kubernetes manifests in `chapter8/gaia-experience/AWorld/env/` to deploy multiple replicas of the **execution‑tools**, **perception‑tools**, or **collaboration‑tools** pods independently from the model server. This separation enables you to handle high tool invocation loads without over‑provisioning expensive GPU resources for the LLM.

### What monitoring options are available for production deployments?

By default, each container logs to stdout, which you can stream using `docker-compose logs -f`. For production monitoring, you can attach Prometheus exporters to the containers or forward logs to cloud logging services. The Model‑as‑a‑Service layer ([`server.py`](https://github.com/bojieli/ai-agent-book/blob/main/server.py) or [`ollama_native.py`](https://github.com/bojieli/ai-agent-book/blob/main/ollama_native.py)) exposes health endpoints at `/health` that return HTTP 200 when the service is ready to accept requests, compatible with Kubernetes liveness and readiness probes.