How to Deploy AgentScope Agents as a Serverless Service or on Kubernetes

You can deploy AgentScope agents to serverless platforms like Google Cloud Run or AWS Lambda, as well as Kubernetes clusters, by packaging your agent code with the official agentscope-runtime Docker image.

AgentScope is an open-source multi-agent framework (agentscope-ai/agentscope) that lets you build AI agents in pure Python. Because the SDK ships as a containerized runtime, you can deploy AgentScope agents to any cloud platform that supports containers, from fully managed serverless environments to self-managed Kubernetes clusters.

Architecture Overview

AgentScope deployments consist of four core components working together to expose your agent via HTTP:

• Agent code: Your custom Python script that imports agentscope and defines a run() entry point. The repository provides a ready-to-use example in examples/deployment/planning_agent/main.py that demonstrates the standard structure expected by the runtime.
• agentscope-runtime image: A lightweight Docker image based on Python 3.11 that pre-installs the agentscope package, selected LLM providers (OpenAI, Ollama), and optional tooling like vector stores or TTS. The image is available on Docker Hub as agentscope/agentscope-runtime:latest.
• Container orchestrator: The platform that executes the container. Serverless platforms like AWS Lambda (via container images), Google Cloud Run, or Azure Container Apps start the container on demand, while Kubernetes runs it inside a pod behind a Service.
• Ingress / API gateway: Exposes the agent’s HTTP endpoint (/run by default). Serverless platforms handle this automatically; in Kubernetes you configure an Ingress resource (NGINX, GKE-Ingress, etc.).

The request flow is straightforward: an HTTP request hits the Ingress, which routes to the container running the agentscope-runtime, which invokes your agent’s run() function and returns the response. The runtime also supports Server-Sent Events for streaming output, as validated in the tests/realtime_*_test.py files.

Deploying to Serverless Platforms

Serverless deployment eliminates infrastructure management by running your container only when requests arrive. The following example uses Google Cloud Run, but the same Dockerfile works for AWS Lambda container images or Azure Container Apps.

Step 1: Containerize Your Agent

Create a Dockerfile that copies your agent code into the runtime image:

FROM agentscope/agentscope-runtime:latest

WORKDIR /app
COPY examples/deployment/planning_agent/main.py .
ENTRYPOINT ["python", "main.py"]

The runtime expects an entrypoint that exposes a run function decorated with @agentscope.agent or @agentscope.tool.

Step 2: Build and Push

Build the image and push it to a container registry:

docker build -t gcr.io/<PROJECT_ID>/planning-agent:latest .
docker push gcr.io/<PROJECT_ID>/planning-agent:latest

Step 3: Deploy to Cloud Run

Deploy the image to Cloud Run to automatically create an HTTPS endpoint:

gcloud run deploy planning-agent \
    --image gcr.io/<PROJECT_ID>/planning-agent:latest \
    --platform managed \
    --region us-central1 \
    --allow-unauthenticated

Cloud Run automatically scales to zero when idle and spins up instances on demand. The endpoint accepts JSON payloads matching your agent’s input schema as defined by the decorators in your source code.

Deploying to Kubernetes

For persistent workloads or complex multi-agent systems requiring sidecars (vector databases, tool sandboxes), Kubernetes provides fine-grained control over scaling and networking.

Step 1: Create the Deployment

Define a Deployment that runs your agent container. Reference the GitHub Container Registry image ghcr.io/agentscope-ai/agentscope-runtime:latest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: planning-agent
spec:
  replicas: 2
  selector:
    matchLabels:
      app: planning-agent
  template:
    metadata:
      labels:
        app: planning-agent
    spec:
      containers:
        - name: agent
          image: ghcr.io/agentscope-ai/agentscope-runtime:latest
          command: ["python", "main.py"]
          env:
            - name: AGENTSCOPE_LOG_LEVEL
              value: "INFO"

Step 2: Expose with a Service

Create a Service to route traffic to the pods on port 8000, the default agentscope-runtime HTTP port:

apiVersion: v1
kind: Service
metadata:
  name: planning-agent-svc
spec:
  selector:
    app: planning-agent
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8000

Step 3: Configure Ingress

Add an Ingress resource to terminate TLS and route external traffic:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: planning-agent-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: agentscope.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: planning-agent-svc
                port:
                  number: 80

Apply the manifests:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

Your agent is now reachable at https://agentscope.example.com/run. Scale horizontally with kubectl scale deployment planning-agent --replicas=5 or add a HorizontalPodAutoscaler for automatic scaling based on CPU or request latency.

Production Best Practices

Deploying AgentScope agents to production requires attention to security, observability, and performance.

• Secret handling: Store LLM API keys and database credentials as Kubernetes Secrets or mounted secrets in Cloud Run. Never hard-code credentials in your agent source code.
• Observability: Enable OpenTelemetry tracing by setting the OTEL_SERVICE_NAME environment variable. The runtime automatically emits traces for each request, as demonstrated in docs/tutorial/en/src/task_tracing.py.
• Low-latency LLM backends: When using vLLM or OpenAI’s streaming API, configure the runtime with --enable-auto-tool-choice and --tool-call-parser flags documented in docs/tutorial/en/src/task_model.py.
• Tool sandboxing: For agents executing arbitrary tools (browsers, VNC), use the agentscope-runtime sandbox that isolates processes in separate user namespaces. Examples are available in examples/functionality/vector_store/ and examples/functionality/tts/.
• Graceful shutdown: The runtime listens for SIGTERM and finishes in-flight requests before exiting, ensuring zero-downtime rolling updates in Kubernetes.

Summary

• AgentScope agents are standard Python programs that run inside the agentscope-runtime container image.
• You can deploy to serverless platforms (Cloud Run, AWS Lambda) by building a custom image and pushing it to a registry.
• Kubernetes deployments use standard Deployment, Service, and Ingress manifests, with the runtime listening on port 8000.
• Production deployments require proper secret management, OpenTelemetry tracing, and graceful shutdown handling for high availability.

Frequently Asked Questions

Can I deploy AgentScope agents on AWS Lambda?

Yes. AWS Lambda supports container images up to 10 GB. Build your image using the agentscope-runtime base, push it to Amazon ECR, and configure the Lambda function to use the container image instead of a ZIP deployment. Ensure your run() handler conforms to Lambda’s invocation interface or use an adapter like aws-lambda-web-adapter for HTTP-mode functions.

How do I handle scaling for high-traffic agent workloads?

For serverless platforms, scaling is automatic based on concurrent request limits you configure. In Kubernetes, use a HorizontalPodAutoscaler targeting CPU or custom metrics (request latency). The runtime is stateless, so you can safely run multiple replicas behind a load balancer. For stateful agents requiring shared memory, deploy a Redis sidecar or use a centralized vector store like Milvus (examples in examples/functionality/vector_store/).

Can I use local LLMs like Ollama when deploying to the cloud?

Yes. Include Ollama in your container image or deploy it as a separate sidecar container in the same Kubernetes pod. Configure your agent to point to localhost:11434 (or the appropriate service name if using a sidecar pattern). For serverless deployments, consider using vLLM with the flags documented in docs/tutorial/en/src/task_model.py for efficient GPU utilization.

What is the difference between serverless and Kubernetes deployment for AgentScope?

Serverless (Cloud Run, Lambda) is optimal for event-driven, bursty traffic with cost-effective scale-to-zero behavior, but imposes request timeout limits (e.g., 60 minutes for Cloud Run, 15 minutes for Lambda). Kubernetes is better for long-running agents, persistent connections (WebSockets, Server-Sent Events), and complex topologies requiring sidecars for vector databases or tool sandboxes. Both use the same agentscope-runtime image and agent code.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}