What Are the Main Components of Agent Substrate?

Agent Substrate consists of three architectural layers—Control Plane (ate-api-server), Node Supervisor (atelet/ateom), and Data Plane (WorkerPool CRD, ActorTemplate CRD, atenet)—that together enable high-density, sub-100ms activation of idle actors on Kubernetes.

Agent Substrate is an open-source runtime layer built atop Kubernetes designed to manage billions of idle "actors" with minimal resource overhead. According to the agent-substrate/substrate repository, the system separates concerns across a Control Plane that orchestrates state, a Node Supervisor that manages local sandboxes, and a Data Plane that handles routing and workload definitions.

Control Plane Layer

The Control Plane acts as the central brain of Agent Substrate, exposing gRPC and CLI interfaces for actor lifecycle management. As defined in docs/architecture.md (lines 300-312), this layer maintains authoritative state and scheduling decisions.

ate-api-server

The ate-api-server is the primary component exposing the public API. It hosts the State Store (backed by Redis) that maintains actor-to-worker mappings and snapshot metadata. The server also contains the Scheduler, which selects ready workers for incoming actor assignments, and the Workflow Engine, which orchestrates the multi-step resume and suspend sequences.

Key API methods include ResumeActor and SuspendActor, defined in pkg/proto/ateapipb/ateapi.pb.go, which clients invoke to transition actors between idle and active states.

State Persistence Strategy

High-frequency state—such as actor locations and session tokens—lives in the fast Redis store, while bulky snapshot blobs reside in object storage (GCS/S3) via the Storage Mover. This dual-layer persistence model allows the system to track billions of actors while maintaining low-latency operations.

Node Supervisor Layer

Running on every cluster node as a DaemonSet, the Node Supervisor manages the physical execution environment. As documented in docs/architecture.md (lines 313-321), this layer comprises two distinct processes.

atelet Daemon

atelet supervises the pool of physical worker pods on a node. It streams snapshots to and from durable storage, reports node capacity to the Control Plane, and manages the lifecycle of the sandbox runtime. The volume management logic is implemented in internal/volume/csi/plugin.go and internal/volume/csi/controller.go, which handle snapshot creation and restoration.

ateom Sandbox-Herder

Inside each worker pod, ateom communicates directly with the sandbox runtime—either gVisor or Kata micro-VMs—to execute three critical operations: RunWorkload, CheckpointWorkload, and RestoreWorkload. This component ensures that actor state is captured cleanly during suspension and rehydrated correctly during resume.

Data Plane Layer

The Data Plane defines the Kubernetes Custom Resource Definitions (CRDs) and networking infrastructure that enable actor discovery and communication.

WorkerPool and ActorTemplate CRDs

The WorkerPool CRD defines a fleet of pre-started "warm" pods that remain idle awaiting assignments. These pools ensure that activation latency stays under 100 milliseconds by eliminating container startup time. The ActorTemplate CRD, listed in pkg/client/listers/api/v1alpha1/actortemplate.go, describes an immutable snapshot of an actor including its container image, configuration, and environment variables.

Sandbox Configuration

SandboxConfig supplies the actual sandbox binaries (gVisor or Kata Containers) that workers use to isolate actor workloads. This abstraction allows operators to swap isolation mechanisms without changing higher-level workload definitions.

atenet Router and atunnel

atenet provides actor-aware networking. A DNS mesh resolves addresses in the format actor.<atespace>.actors.resources.substrate.ate.dev. When a request arrives, the router queries the Control Plane to locate or resume the target actor, then establishes an mTLS tunnel called atunnel to the selected worker. Implementation details are available in cmd/atenet/README.md and the demo client at demos/sandbox/client/main.go.

Component Integration Workflow

The Agent Substrate components orchestrate the following lifecycle sequence:

1. Actor Creation: A user or automation invokes the Control Plane API to create an Actor record pointing to an ActorTemplate.
2. Idle Waiting: A WorkerPool maintains warm pods on nodes. Each pod runs atelet and an ateom container for the configured sandbox class.
3. Request Arrival: The atenet DNS router receives a client request, extracts the actor identifier, and queries the Control Plane to trigger ResumeActor.
4. Scheduling: The Scheduler selects an idle worker, and atelet streams the latest snapshot from object storage into the worker's sandbox via ateom.
5. Traffic Forwarding: atenet opens an mTLS atunnel to the worker pod and forwards the request to the now-running actor.
6. Suspension: When the actor becomes idle, the Control Plane instructs atelet and ateom to checkpoint the sandbox, store the snapshot, and return the worker to the pool.

Implementation Examples

Go Client for Actor Lifecycle

The following example demonstrates connecting to ate-api-server to manage an actor's lifecycle and issue commands through atenet:

package main

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"syscall"

	"github.com/agent-substrate/substrate/internal/resources"
	"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
	"github.com/spf13/pflag"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
)

type ProcessRequest struct {
	Command []string          `json:"command"`
	EnvVars map[string]string `json:"envvars,omitempty"`
}
type ProcessResponse struct {
	Stdout string `json:"stdout"`
	Stderr string `json:"stderr"`
	Error  string `json:"error,omitempty"`
}

// dialAteAPI connects to the Control Plane (ate-api-server)
func dialAteAPI(endpoint string) (ateapipb.ControlClient, *grpc.ClientConn, error) {
	creds := credentials.NewTLS(&tls.Config{InsecureSkipVerify: true})
	conn, err := grpc.NewClient(endpoint, grpc.WithTransportCredentials(creds))
	if err != nil {
		return nil, nil, err
	}
	return ateapipb.NewControlClient(conn), conn, nil
}

func main() {
	actorName := pflag.String("name", "", "Actor name (required)")
	atespace := pflag.String("atespace", "", "Atespace (required)")
	ateapi := pflag.String("ateapi", "localhost:8080", "Control-plane address")
	atenet := pflag.String("atenet", "localhost:8000", "HTTP router address")
	pflag.Parse()

	if *actorName == "" || *atespace == "" {
		log.Fatal("both --name and --atespace are required")
	}
	actor := resources.ActorRef{Atespace: *atespace, Name: *actorName}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	sig := make(chan os.Signal, 1)
	signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
	go func() { <-sig; cancel() }()

	// Resume the actor via Control Plane
	client, conn, err := dialAteAPI(*ateapi)
	if err != nil {
		log.Fatalf("dial ateapi: %v", err)
	}
	defer conn.Close()
	if _, err = client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actor.ToObjectRef()}); err != nil {
		log.Fatalf("resume actor: %v", err)
	}
	fmt.Println("Actor resumed – you can now issue commands.")

	// Ensure suspension on exit
	defer func() {
		if _, err = client.SuspendActor(context.Background(),
			&ateapipb.SuspendActorRequest{Actor: actor.ToObjectRef()}); err != nil {
			log.Printf("suspend error: %v", err)
		} else {
			fmt.Println("\nActor suspended.")
		}
	}()

	// REPL sending commands through atenet router
	scanner := bufio.NewScanner(os.Stdin)
	for {
		fmt.Print("sandbox> ")
		if !scanner.Scan() {
			break
		}
		line := strings.TrimSpace(scanner.Text())
		if line == "" {
			continue
		}
		if line == "exit" {
			break
		}
		resp, err := runCommand(ctx, *atenet, actor, line)
		if err != nil {
			fmt.Printf("error: %v\n", err)
			continue
		}
		if resp.Error != "" {
			fmt.Printf("cmd error: %s\n", resp.Error)
		}
		fmt.Print(resp.Stdout)
		fmt.Print(resp.Stderr)
	}
}

func runCommand(ctx context.Context, atenetAddr string, actor resources.ActorRef, cmd string) (*ProcessResponse, error) {
	url := fmt.Sprintf("http://%s/process", atenetAddr)
	body, _ := json.Marshal(ProcessRequest{Command: []string{"sh", "-c", cmd}})
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewBuffer(body))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-Type", "application/json")
	req.Host = resources.ActorDNSName(actor) // Triggers actor-aware routing
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		b, _ := io.ReadAll(resp.Body)
		return nil, fmt.Errorf("status %d: %s", resp.StatusCode, string(b))
	}
	var pr ProcessResponse
	if err = json.NewDecoder(resp.Body).Decode(&pr); err != nil {
		return nil, err
	}
	return &pr, nil
}

This client demonstrates connecting to ate-api-server for lifecycle control and routing traffic through atenet using actor-aware DNS names.

kubectl-ate CLI Usage

The kubectl-ate binary, implemented in cmd/kubectl-ate/main.go, provides a command-line interface wrapping the same gRPC methods:

# List actors in an atespace

kubectl-ate --atespace=demo list actors

# Create an actor from a template

kubectl-ate --atespace=demo create actor my-app --template=hello-tmpl

# Manually resume an actor (normally triggered automatically by atenet)

kubectl-ate --atespace=demo resume actor my-app

Summary

• Agent Substrate employs a three-layer architecture: Control Plane, Node Supervisor, and Data Plane.
• The Control Plane (ate-api-server) exposes gRPC APIs for actor management and hosts the Redis-based State Store and Scheduler.
• The Node Supervisor runs atelet (node-level daemon) and ateom (per-pod sandbox controller) to handle CheckpointWorkload and RestoreWorkload operations.
• The Data Plane uses WorkerPool and ActorTemplate CRDs to define warm capacity and immutable actor snapshots.
• atenet provides DNS-based service discovery and mTLS tunneling (atunnel) to route requests to active actors.
• Source implementations reside in pkg/proto/ateapipb/ateapi.pb.go (API definitions), internal/volume/csi/ (snapshot management), and cmd/atenet/ (networking stack).

Frequently Asked Questions

What is the role of ate-api-server in Agent Substrate?

The ate-api-server functions as the Control Plane brain exposed via gRPC in pkg/proto/ateapipb/ateapi.pb.go. It hosts the State Store (Redis), the Scheduler for worker assignment, and the Workflow Engine that orchestrates resume/suspend sequences. All actor lifecycle operations, including ResumeActor and SuspendActor, flow through this component.

How does atenet route traffic to actors?

atenet implements a DNS mesh that resolves actor addresses in the format actor.<atespace>.actors.resources.substrate.ate.dev. When a request arrives, the router queries the Control Plane to ensure the actor is active, potentially triggering a resume, then establishes an mTLS atunnel to the target worker pod. This mechanism is detailed in cmd/atenet/README.md and demonstrated in demos/sandbox/client/main.go.

What sandbox runtimes does Agent Substrate support?

Agent Substrate supports gVisor and Kata Containers (micro-VMs) through the SandboxConfig abstraction. The ateom component inside each worker pod communicates with these runtimes to execute RunWorkload, CheckpointWorkload, and RestoreWorkload operations, providing isolation while maintaining fast activation times.

How does Agent Substrate achieve sub-100ms activation latency?

The system maintains WorkerPools—pre-started pods that remain idle and warm—eliminating container startup overhead. When a request arrives, atelet streams only the snapshot delta from object storage (GCS/S3) into the existing sandbox via ateom, rather than starting a new container. This architecture, combined with the Redis-based State Store for fast lookups, enables activation in under 100 milliseconds.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}