# What Are the Main Components of Agent Substrate?

> Explore the main components of Agent Substrate: Control Plane, Node Supervisor, and Data Plane. Understand how these layers facilitate rapid actor activation on Kubernetes.

- Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate)
- Tags: getting-started
- Published: 2026-08-22

---

**Agent Substrate consists of three architectural layers—Control Plane (ate-api-server), Node Supervisor (atelet/ateom), and Data Plane (WorkerPool CRD, ActorTemplate CRD, atenet)—that together enable high-density, sub-100ms activation of idle actors on Kubernetes.**

Agent Substrate is an open-source runtime layer built atop Kubernetes designed to manage billions of idle "actors" with minimal resource overhead. According to the `agent-substrate/substrate` repository, the system separates concerns across a Control Plane that orchestrates state, a Node Supervisor that manages local sandboxes, and a Data Plane that handles routing and workload definitions.

## Control Plane Layer

The **Control Plane** acts as the central brain of Agent Substrate, exposing gRPC and CLI interfaces for actor lifecycle management. As defined in [`docs/architecture.md`](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md) (lines 300-312), this layer maintains authoritative state and scheduling decisions.

### ate-api-server

The **`ate-api-server`** is the primary component exposing the public API. It hosts the **State Store** (backed by Redis) that maintains actor-to-worker mappings and snapshot metadata. The server also contains the **Scheduler**, which selects ready workers for incoming actor assignments, and the **Workflow Engine**, which orchestrates the multi-step resume and suspend sequences.

Key API methods include `ResumeActor` and `SuspendActor`, defined in [`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go), which clients invoke to transition actors between idle and active states.

### State Persistence Strategy

High-frequency state—such as actor locations and session tokens—lives in the fast Redis store, while bulky snapshot blobs reside in object storage (GCS/S3) via the **Storage Mover**. This dual-layer persistence model allows the system to track billions of actors while maintaining low-latency operations.

## Node Supervisor Layer

Running on every cluster node as a DaemonSet, the **Node Supervisor** manages the physical execution environment. As documented in [`docs/architecture.md`](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md) (lines 313-321), this layer comprises two distinct processes.

### atelet Daemon

**`atelet`** supervises the pool of physical worker pods on a node. It streams snapshots to and from durable storage, reports node capacity to the Control Plane, and manages the lifecycle of the sandbox runtime. The volume management logic is implemented in [`internal/volume/csi/plugin.go`](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/plugin.go) and [`internal/volume/csi/controller.go`](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/controller.go), which handle snapshot creation and restoration.

### ateom Sandbox-Herder

Inside each worker pod, **`ateom`** communicates directly with the sandbox runtime—either gVisor or Kata micro-VMs—to execute three critical operations: `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload`. This component ensures that actor state is captured cleanly during suspension and rehydrated correctly during resume.

## Data Plane Layer

The **Data Plane** defines the Kubernetes Custom Resource Definitions (CRDs) and networking infrastructure that enable actor discovery and communication.

### WorkerPool and ActorTemplate CRDs

The **`WorkerPool`** CRD defines a fleet of pre-started "warm" pods that remain idle awaiting assignments. These pools ensure that activation latency stays under 100 milliseconds by eliminating container startup time. The **`ActorTemplate`** CRD, listed in [`pkg/client/listers/api/v1alpha1/actortemplate.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/client/listers/api/v1alpha1/actortemplate.go), describes an immutable snapshot of an actor including its container image, configuration, and environment variables.

### Sandbox Configuration

**`SandboxConfig`** supplies the actual sandbox binaries (gVisor or Kata Containers) that workers use to isolate actor workloads. This abstraction allows operators to swap isolation mechanisms without changing higher-level workload definitions.

### atenet Router and atunnel

**`atenet`** provides actor-aware networking. A DNS mesh resolves addresses in the format `actor.<atespace>.actors.resources.substrate.ate.dev`. When a request arrives, the router queries the Control Plane to locate or resume the target actor, then establishes an mTLS tunnel called **`atunnel`** to the selected worker. Implementation details are available in [`cmd/atenet/README.md`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/README.md) and the demo client at [`demos/sandbox/client/main.go`](https://github.com/agent-substrate/substrate/blob/main/demos/sandbox/client/main.go).

## Component Integration Workflow

The Agent Substrate components orchestrate the following lifecycle sequence:

1. **Actor Creation**: A user or automation invokes the Control Plane API to create an `Actor` record pointing to an `ActorTemplate`.
2. **Idle Waiting**: A `WorkerPool` maintains warm pods on nodes. Each pod runs `atelet` and an `ateom` container for the configured sandbox class.
3. **Request Arrival**: The `atenet` DNS router receives a client request, extracts the actor identifier, and queries the Control Plane to trigger `ResumeActor`.
4. **Scheduling**: The Scheduler selects an idle worker, and `atelet` streams the latest snapshot from object storage into the worker's sandbox via `ateom`.
5. **Traffic Forwarding**: `atenet` opens an mTLS `atunnel` to the worker pod and forwards the request to the now-running actor.
6. **Suspension**: When the actor becomes idle, the Control Plane instructs `atelet` and `ateom` to checkpoint the sandbox, store the snapshot, and return the worker to the pool.

## Implementation Examples

### Go Client for Actor Lifecycle

The following example demonstrates connecting to `ate-api-server` to manage an actor's lifecycle and issue commands through `atenet`:

```go
package main

import (
	"bufio"
	"bytes"
	"context"
	"encoding/json"
	"fmt"
	"io"
	"log"
	"net/http"
	"os"
	"os/signal"
	"strings"
	"syscall"

	"github.com/agent-substrate/substrate/internal/resources"
	"github.com/agent-substrate/substrate/pkg/proto/ateapipb"
	"github.com/spf13/pflag"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials"
)

type ProcessRequest struct {
	Command []string          `json:"command"`
	EnvVars map[string]string `json:"envvars,omitempty"`
}
type ProcessResponse struct {
	Stdout string `json:"stdout"`
	Stderr string `json:"stderr"`
	Error  string `json:"error,omitempty"`
}

// dialAteAPI connects to the Control Plane (ate-api-server)
func dialAteAPI(endpoint string) (ateapipb.ControlClient, *grpc.ClientConn, error) {
	creds := credentials.NewTLS(&tls.Config{InsecureSkipVerify: true})
	conn, err := grpc.NewClient(endpoint, grpc.WithTransportCredentials(creds))
	if err != nil {
		return nil, nil, err
	}
	return ateapipb.NewControlClient(conn), conn, nil
}

func main() {
	actorName := pflag.String("name", "", "Actor name (required)")
	atespace := pflag.String("atespace", "", "Atespace (required)")
	ateapi := pflag.String("ateapi", "localhost:8080", "Control-plane address")
	atenet := pflag.String("atenet", "localhost:8000", "HTTP router address")
	pflag.Parse()

	if *actorName == "" || *atespace == "" {
		log.Fatal("both --name and --atespace are required")
	}
	actor := resources.ActorRef{Atespace: *atespace, Name: *actorName}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	sig := make(chan os.Signal, 1)
	signal.Notify(sig, os.Interrupt, syscall.SIGTERM)
	go func() { <-sig; cancel() }()

	// Resume the actor via Control Plane
	client, conn, err := dialAteAPI(*ateapi)
	if err != nil {
		log.Fatalf("dial ateapi: %v", err)
	}
	defer conn.Close()
	if _, err = client.ResumeActor(ctx, &ateapipb.ResumeActorRequest{Actor: actor.ToObjectRef()}); err != nil {
		log.Fatalf("resume actor: %v", err)
	}
	fmt.Println("Actor resumed – you can now issue commands.")

	// Ensure suspension on exit
	defer func() {
		if _, err = client.SuspendActor(context.Background(),
			&ateapipb.SuspendActorRequest{Actor: actor.ToObjectRef()}); err != nil {
			log.Printf("suspend error: %v", err)
		} else {
			fmt.Println("\nActor suspended.")
		}
	}()

	// REPL sending commands through atenet router
	scanner := bufio.NewScanner(os.Stdin)
	for {
		fmt.Print("sandbox> ")
		if !scanner.Scan() {
			break
		}
		line := strings.TrimSpace(scanner.Text())
		if line == "" {
			continue
		}
		if line == "exit" {
			break
		}
		resp, err := runCommand(ctx, *atenet, actor, line)
		if err != nil {
			fmt.Printf("error: %v\n", err)
			continue
		}
		if resp.Error != "" {
			fmt.Printf("cmd error: %s\n", resp.Error)
		}
		fmt.Print(resp.Stdout)
		fmt.Print(resp.Stderr)
	}
}

func runCommand(ctx context.Context, atenetAddr string, actor resources.ActorRef, cmd string) (*ProcessResponse, error) {
	url := fmt.Sprintf("http://%s/process", atenetAddr)
	body, _ := json.Marshal(ProcessRequest{Command: []string{"sh", "-c", cmd}})
	req, err := http.NewRequestWithContext(ctx, http.MethodPost, url, bytes.NewBuffer(body))
	if err != nil {
		return nil, err
	}
	req.Header.Set("Content-Type", "application/json")
	req.Host = resources.ActorDNSName(actor) // Triggers actor-aware routing
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		return nil, err
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		b, _ := io.ReadAll(resp.Body)
		return nil, fmt.Errorf("status %d: %s", resp.StatusCode, string(b))
	}
	var pr ProcessResponse
	if err = json.NewDecoder(resp.Body).Decode(&pr); err != nil {
		return nil, err
	}
	return &pr, nil
}

```

This client demonstrates connecting to **`ate-api-server`** for lifecycle control and routing traffic through **`atenet`** using actor-aware DNS names.

### kubectl-ate CLI Usage

The `kubectl-ate` binary, implemented in [`cmd/kubectl-ate/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/kubectl-ate/main.go), provides a command-line interface wrapping the same gRPC methods:

```bash

# List actors in an atespace

kubectl-ate --atespace=demo list actors

# Create an actor from a template

kubectl-ate --atespace=demo create actor my-app --template=hello-tmpl

# Manually resume an actor (normally triggered automatically by atenet)

kubectl-ate --atespace=demo resume actor my-app

```

## Summary

- **Agent Substrate** employs a three-layer architecture: Control Plane, Node Supervisor, and Data Plane.
- The **Control Plane** (`ate-api-server`) exposes gRPC APIs for actor management and hosts the Redis-based State Store and Scheduler.
- The **Node Supervisor** runs `atelet` (node-level daemon) and `ateom` (per-pod sandbox controller) to handle `CheckpointWorkload` and `RestoreWorkload` operations.
- The **Data Plane** uses `WorkerPool` and `ActorTemplate` CRDs to define warm capacity and immutable actor snapshots.
- **atenet** provides DNS-based service discovery and mTLS tunneling (`atunnel`) to route requests to active actors.
- Source implementations reside in [`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go) (API definitions), `internal/volume/csi/` (snapshot management), and `cmd/atenet/` (networking stack).

## Frequently Asked Questions

### What is the role of ate-api-server in Agent Substrate?

The `ate-api-server` functions as the Control Plane brain exposed via gRPC in [`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go). It hosts the State Store (Redis), the Scheduler for worker assignment, and the Workflow Engine that orchestrates resume/suspend sequences. All actor lifecycle operations, including `ResumeActor` and `SuspendActor`, flow through this component.

### How does atenet route traffic to actors?

`atenet` implements a DNS mesh that resolves actor addresses in the format `actor.<atespace>.actors.resources.substrate.ate.dev`. When a request arrives, the router queries the Control Plane to ensure the actor is active, potentially triggering a resume, then establishes an mTLS `atunnel` to the target worker pod. This mechanism is detailed in [`cmd/atenet/README.md`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/README.md) and demonstrated in [`demos/sandbox/client/main.go`](https://github.com/agent-substrate/substrate/blob/main/demos/sandbox/client/main.go).

### What sandbox runtimes does Agent Substrate support?

Agent Substrate supports **gVisor** and **Kata Containers** (micro-VMs) through the `SandboxConfig` abstraction. The `ateom` component inside each worker pod communicates with these runtimes to execute `RunWorkload`, `CheckpointWorkload`, and `RestoreWorkload` operations, providing isolation while maintaining fast activation times.

### How does Agent Substrate achieve sub-100ms activation latency?

The system maintains **WorkerPools**—pre-started pods that remain idle and warm—eliminating container startup overhead. When a request arrives, `atelet` streams only the snapshot delta from object storage (GCS/S3) into the existing sandbox via `ateom`, rather than starting a new container. This architecture, combined with the Redis-based State Store for fast lookups, enables activation in under 100 milliseconds.