How to Deploy AI Agents to Production: A Complete Guide to Model Serving, MCP Tools, and Orchestration

Deploying AI agents to production requires three layers: a model‑as‑a‑service backend (vLLM or Ollama), containerized MCP tool servers for execution and perception, and Docker or Kubernetes orchestration with proper secrets management.

This guide walks you through the production deployment architecture defined in the bojieli/ai-agent-book repository, covering how to serve LLMs locally, run Model‑Context‑Protocol (MCP) tool containers, and orchestrate the entire stack using Docker Compose or Kubernetes.

Architectural Blueprint for Production Deployment

Production deployment of an AI agent—which combines an LLM, context, and tools—depends on a three‑layer infrastructure: model serving, tool back‑ends, and orchestration.

Model‑as‑a‑Service Layer

你 must expose the LLM through an OpenAI‑compatible API. According to the source code in chapter2/local_llm_serving, you have two primary options:

• vLLM – Use this CUDA‑enabled option on Linux GPUs for high‑throughput token generation with KV‑cache reuse.
• Ollama – Use this for macOS, Windows, or Linux without a GPU, pulling lightweight models like qwen3:0.6b.

The main.py wrapper auto‑detects the environment, launches the appropriate server (server.py for vLLM or ollama_native.py for Ollama), and exposes a unified endpoint at POST /v1/chat/completions. This design lets you swap backends without changing agent code, enabling seamless migration from local development (Ollama) to high‑performance GPU clusters (vLLM).

MCP Tool Servers

The repository provides three categories of containerized MCP (Model‑Context‑Protocol) servers, each defined in chapter4/:

• execution‑tools – Sandboxed multi‑language code execution (Python, Node, Go, Java, C++, Rust, PHP, Bash). The Dockerfile installs language‑specific runtimes and mounts an execution-workspace volume for isolation.
• perception‑tools – Document ingestion, web search, OCR, and PDF parsing using Python stacks including tesseract, requests, and BeautifulSoup.
• collaboration‑tools – Browser automation, Excel processing, and human‑in‑the‑loop dialogs using Headless Chromium and openpyxl.

All services read a common .env file for API keys (OpenAI, Google, etc.) and expose the same MCP JSON contract, allowing the Agent to call any tool with a uniform request shape.

Orchestration and Production Concerns

The chapter4/docker-compose.yml file defines the three MCP services with network isolation and shared volumes. For production environments, the repository recommends four critical practices:

1. Secrets Management – Replace the plain‑text .env file with Docker Swarm or Kubernetes secrets before deploying to production.
2. Observability – Each container logs to stdout; use docker-compose logs -f for streaming or attach Prometheus exporters for metrics.
3. Scalability – The chapter8/gaia-experience/AWorld/env/README.md describes Kubernetes manifests for horizontal scaling of each MCP service and the model server independently.
4. Separation of Concerns – The Agent’s runtime never embeds model weights; it only communicates via HTTP to the model endpoint and tool services. This enables independent upgrades of models and tools, supporting continuous delivery.

Step‑by‑Step Deployment Examples

Launch the Full Stack with Docker Compose

To deploy AI agents to production using Docker, execute the following commands from the repository root:

# 1️⃣ Copy example env and fill in your keys

cp .env.example .env

# edit .env → set OPENAI_API_KEY, GOOGLE_API_KEY, …

# 2️⃣ Build and launch all services

docker-compose up -d

# 3️⃣ Verify health

curl -s http://localhost:8000/health

The docker-compose.yml file orchestrates the execution‑tools, perception‑tools, and collaboration‑tools containers, pulling base images and building MCP containers automatically.

Serve Local LLMs with vLLM or Ollama

For Linux GPU environments, start the vLLM server:

cd chapter2/local_llm_serving
pip install -r requirements.txt
python server.py  # starts vLLM on port 8000

Now instantiate the agent:

from main import ToolCallingAgent

agent = ToolCallingAgent()  # auto‑detects the vLLM endpoint

response = agent.chat("Summarize the latest AI safety paper.")
print(response)

The ToolCallingAgent class handles endpoint detection and API compatibility, allowing the same code to work with either vLLM or Ollama backends.

Invoke Tools from the Agent

To trigger tool calls (e.g., weather lookup), use:

from main import ToolCallingAgent

agent = ToolCallingAgent()
for chunk in agent.chat("What is the current temperature in Tokyo?", stream=True):
    if chunk["type"] == "tool_result":
        print("🛠️ Tool output:", chunk["content"])

The request flow follows this path:

• Agent → model (/v1/chat/completions)
• LLM decides to call get_current_temperature
• MCP perception‑tools receives a JSON RPC call, fetches data from Open‑Meteo, and returns the result

The plumbing logic resides in chapter4/perception-tools/src/main.py.

Scale with Kubernetes

For horizontal scaling, apply the Kubernetes manifests from chapter8/gaia-experience/AWorld/env/:

kubectl apply -f k8s/model-server.yaml
kubectl apply -f k8s/execution-tools.yaml
kubectl apply -f k8s/perception-tools.yaml
kubectl apply -f k8s/collaboration-tools.yaml

Each manifest references Docker images built from the Chapter 4 Dockerfiles. The README.md in chapter8/gaia-experience/AWorld/env/ details the required secret objects and ingress configuration for production traffic.

Critical Source Files and References

File	Purpose
chapter2/local_llm_serving/README.md	Guide to running portable LLM servers and using the Python SDK.
chapter4/DOCKER_DEPLOYMENT.md	Step‑by‑step Docker deployment instructions with security advice.
chapter4/docker-compose.yml	Single‑file composition of all three MCP services.
chapter4/execution-tools/Dockerfile	Multi‑language sandbox environment for code execution.
chapter4/perception-tools/Dockerfile	Document processing and web search capabilities.
chapter4/collaboration-tools/Dockerfile	Browser automation and Excel processing tools.
chapter8/gaia-experience/AWorld/env/README.md	Production Kubernetes deployment and scaling strategies.
chapter8/gaia-experience/AWorld/aworlddistributed/README.md	Distributed MCP server architecture for load‑balanced deployments.

Summary

• Model Serving: Use chapter2/local_llm_serving to expose vLLM or Ollama via an OpenAI‑compatible API, enabling backend swaps without code changes.
• Tool Infrastructure: Deploy execution, perception, and collaboration MCP servers using chapter4/docker-compose.yml or Kubernetes manifests from Chapter 8.
• Security: Replace .env files with proper secret management (Docker Swarm or Kubernetes secrets) before production deployment.
• Scalability: The decoupled architecture allows independent scaling of LLM inference and tool execution through container orchestration.
• Integration: The ToolCallingAgent class provides a unified Python interface that routes LLM calls and tool invocations to the appropriate back‑ends automatically.

Frequently Asked Questions

What is the difference between vLLM and Ollama for production deployment?

vLLM is optimized for CUDA‑enabled Linux servers and provides high‑throughput token generation with KV‑cache reuse, making it ideal for GPU clusters. Ollama runs on macOS, Windows, and Linux without requiring a GPU and pulls lightweight quantized models, making it suitable for development environments or edge deployment. The main.py wrapper in chapter2/local_llm_serving automatically detects which backend is available and exposes a uniform OpenAI‑compatible API.

How do MCP tool servers handle security and isolation?

Each tool category runs in its own Docker container defined in separate Dockerfiles within chapter4/. The execution‑tools container uses a dedicated execution-workspace volume to sandbox multi‑language code execution (Python, Node, Go, Java, etc.), while perception‑tools and collaboration‑tools run isolated processes for document parsing and browser automation. All containers read secrets from environment variables injected via .env files or Kubernetes secret objects, ensuring API keys never reside in the application code.

Can I scale individual components independently?

Yes. The architecture decouples the LLM inference layer from the tool execution layer, allowing you to scale each horizontally. Use the Kubernetes manifests in chapter8/gaia-experience/AWorld/env/ to deploy multiple replicas of the execution‑tools, perception‑tools, or collaboration‑tools pods independently from the model server. This separation enables you to handle high tool invocation loads without over‑provisioning expensive GPU resources for the LLM.

What monitoring options are available for production deployments?

By default, each container logs to stdout, which you can stream using docker-compose logs -f. For production monitoring, you can attach Prometheus exporters to the containers or forward logs to cloud logging services. The Model‑as‑a‑Service layer (server.py or ollama_native.py) exposes health endpoints at /health that return HTTP 200 when the service is ready to accept requests, compatible with Kubernetes liveness and readiness probes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}