# How the Control Plane Is Implemented in Agent Substrate: A Deep Dive into the Three-Tier Architecture

> Explore the three-tier architecture of Agent Substrate's control plane. Learn how ateapi, atecontroller, and atelet manage millions of actor sandboxes with CRDs and reconcile loops.

- Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate)
- Tags: deep-dive
- Published: 2026-08-22

---

**Agent Substrate implements its control plane as a distributed three-tier system comprising the `ateapi` gRPC server, the `atecontroller` Kubernetes operator, and the `atelet` node agent, leveraging etcd-backed CRDs and controller-runtime reconcile loops to manage millions of ephemeral actor sandboxes.**

The [agent-substrate/substrate](https://github.com/agent-substrate/substrate) repository defines a lightweight, high-performance control plane purpose-built for orchestrating short-lived, sandboxed workloads called *actors*. Unlike traditional orchestrators that handle heavy infrastructure provisioning, Agent Substrate delegates infrastructure concerns to Kubernetes while maintaining a focused control plane for rapid actor lifecycle operations.

## Control Plane Architecture Overview

The control plane consists of three core binaries that communicate via gRPC and persist state through the Kubernetes API server:

- **`ateapi`** – The central gRPC API server that exposes endpoints for actor management
- **`atecontroller`** – A Kubernetes operator that watches custom resources and reconciles desired state
- **`atelet`** – A node-level agent that executes lifecycle commands on worker nodes

All components share a unified data model defined in Protocol Buffers and stored as Custom Resource Definitions (CRDs) in etcd via the Kubernetes API server, as detailed in [[`docs/architecture.md`](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md)](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md).

## The API Server (ateapi)

The `ateapi` binary serves as the entry point for control plane operations. It exposes a gRPC interface defined in [[`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go)](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go).

### gRPC Interface Definition

The protobuf definitions in [`ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/ateapi.pb.go) generate the Go structs and service interfaces used for all control plane communication. These definitions cover actor lifecycle operations including creation, suspension, resumption, and termination.

```go
// Conceptual usage based on ateapi.pb.go generated types
client := ateapipb.NewActorManagementClient(conn)
resp, err := client.CreateActor(ctx, &ateapipb.CreateActorRequest{
    TemplateRef: "actor-template-001",
    WorkerPool:  "pool-us-east-1",
})

```

### Server Implementation

Located in `cmd/ateapi`, the server initializes the gRPC listener and registers the service implementations. It translates incoming gRPC calls into Kubernetes API operations, creating or updating CRD instances that represent the desired actor state.

## The Kubernetes Controller (atecontroller)

The `atecontroller` binary implements the reconciliation logic that bridges declarative CRD state with actual worker node capacity. It uses the [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) framework to watch resources and trigger reconcile loops.

### CRD Reconciliation Logic

The primary controllers reside in `cmd/atecontroller/internal/controllers/`:

- **[[`workerpool_controller.go`](https://github.com/agent-substrate/substrate/blob/main/workerpool_controller.go)](https://github.com/agent-substrate/substrate/blob/main/cmd/atecontroller/internal/controllers/workerpool_controller.go)** – Manages the lifecycle of worker pools, tracking capacity and scheduling decisions
- **[[`actortemplate_controller.go`](https://github.com/agent-substrate/substrate/blob/main/actortemplate_controller.go)](https://github.com/agent-substrate/substrate/blob/main/cmd/atecontroller/internal/controllers/actortemplate_controller.go)** – Handles actor template versioning and validation

```go
// Conceptual reconcile loop pattern as implemented in workerpool_controller.go
func (r *WorkerPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var pool substratev1.WorkerPool
    if err := r.Get(ctx, req.NamespacedName, &pool); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    // Update pool status based on available atelet nodes
    pool.Status.AvailableCapacity = r.calculateCapacity(ctx, &pool)
    return ctrl.Result{RequeueAfter: 30 * time.Second}, r.Status().Update(ctx, &pool)
}

```

### State Management via etcd

The controller persists all state through the Kubernetes API server, storing CRDs (Actor, ActorTemplate, WorkerPool) in etcd. This design leverages Kubernetes' distributed storage without requiring a separate database for the control plane.

## The Node Agent (atelet)

Running on each worker node, the `atelet` binary receives gRPC commands from the `atecontroller` and manages the actual sandbox processes (`ateom` instances).

### gRPC Command Execution

The `atelet` exposes a gRPC server that accepts lifecycle commands:

- **StartActor** – Mounts volume snapshots and launches the `ateom` sandbox
- **SuspendActor** – Checkpoints process state and uploads to object storage
- **ResumeActor** – Restores checkpoint and resumes execution

### CSI Volume Integration

The `atelet` interacts with container storage through the CSI client implementation in [[`internal/volume/csi/client.go`](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/client.go)](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/client.go):

```go
// Volume mounting pattern from internal/volume/csi/client.go
func (c *CSIClient) MountSnapshot(ctx context.Context, snapshotID, targetPath string) error {
    req := &csi.NodePublishVolumeRequest{
        VolumeId:          snapshotID,
        TargetPath:        targetPath,
        VolumeCapability:  c.getVolumeCapability(),
        Readonly:          true,
    }
    _, err := c.nodeClient.NodePublishVolume(ctx, req)
    return err
}

```

## Data Model and Protocol Buffers

The control plane uses a dual-purpose data model where Protocol Buffer definitions serve as both the wire format and the source of truth for CRD generation.

The generated Go code in [[`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go)](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go) provides:

- Message structs for gRPC communication between `ateapi`, `atecontroller`, and `atelet`
- JSON tags for Kubernetes CRD serialization
- Validation methods for resource integrity

This approach ensures consistency between the API schema, database storage, and cross-service communication.

## Observability and Testing

The control plane integrates OpenTelemetry for distributed tracing across all three components. The `atecontroller` uses `controller-runtime`'s `envtest` package for integration testing against a local Kubernetes API server without requiring a full cluster, validating the reconcile logic in [`workerpool_controller.go`](https://github.com/agent-substrate/substrate/blob/main/workerpool_controller.go) against ephemeral etcd instances.

## Summary

- **Agent Substrate's control plane** consists of three binaries: `ateapi` (API layer), `atecontroller` (reconciliation), and `atelet` (execution)
- **State persistence** relies on Kubernetes CRDs stored in etcd, accessed via the standard Kubernetes API
- **Communication** occurs over gRPC using Protocol Buffer definitions from [`pkg/proto/ateapipb/ateapi.pb.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go)
- **Controller logic** in [`workerpool_controller.go`](https://github.com/agent-substrate/substrate/blob/main/workerpool_controller.go) and [`actortemplate_controller.go`](https://github.com/agent-substrate/substrate/blob/main/actortemplate_controller.go) manages resource scheduling using controller-runtime
- **Node operations** are handled by `atelet`, which interfaces with CSI storage via [`internal/volume/csi/client.go`](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/client.go)
- **Design philosophy** keeps the control plane focused and lightweight, delegating infrastructure management to Kubernetes while optimizing for high-frequency actor lifecycle operations as defined in [[`docs/glossary.md`](https://github.com/agent-substrate/substrate/blob/main/docs/glossary.md)](https://github.com/agent-substrate/substrate/blob/main/docs/glossary.md)

## Frequently Asked Questions

### What is the role of the ateapi binary in Agent Substrate?

The `ateapi` binary serves as the gRPC API server and primary entry point for the control plane. It validates incoming requests for actor lifecycle management and translates them into Kubernetes CRD operations, persisting the desired state in etcd via the Kubernetes API server.

### How does atecontroller interact with Kubernetes?

The `atecontroller` uses the controller-runtime library to establish watch streams on Custom Resource Definitions (ActorTemplate, Actor, WorkerPool). When resources change, it triggers reconcile loops in controllers like [`workerpool_controller.go`](https://github.com/agent-substrate/substrate/blob/main/workerpool_controller.go) to match the actual cluster state with the desired state defined in the CRDs.

### What is the relationship between atelet and the control plane?

`atelet` acts as the node-local extension of the control plane. While `ateapi` and `atecontroller` manage global state and scheduling decisions, `atelet` executes the actual commands on worker nodes via gRPC, managing `ateom` sandbox processes and handling volume mounts through the CSI client.

### Where is the control plane state stored?

All control plane state persists in **etcd** through the Kubernetes API server. The `ateapi` server creates CRD instances, `atecontroller` reads and updates their status, and `atelet` reports node-level metrics back to the controller, creating a unified source of truth for the entire system.