How the Control Plane Is Implemented in Agent Substrate: A Deep Dive into the Three-Tier Architecture

Agent Substrate implements its control plane as a distributed three-tier system comprising the ateapi gRPC server, the atecontroller Kubernetes operator, and the atelet node agent, leveraging etcd-backed CRDs and controller-runtime reconcile loops to manage millions of ephemeral actor sandboxes.

The agent-substrate/substrate repository defines a lightweight, high-performance control plane purpose-built for orchestrating short-lived, sandboxed workloads called actors. Unlike traditional orchestrators that handle heavy infrastructure provisioning, Agent Substrate delegates infrastructure concerns to Kubernetes while maintaining a focused control plane for rapid actor lifecycle operations.

Control Plane Architecture Overview

The control plane consists of three core binaries that communicate via gRPC and persist state through the Kubernetes API server:

• ateapi – The central gRPC API server that exposes endpoints for actor management
• atecontroller – A Kubernetes operator that watches custom resources and reconciles desired state
• atelet – A node-level agent that executes lifecycle commands on worker nodes

All components share a unified data model defined in Protocol Buffers and stored as Custom Resource Definitions (CRDs) in etcd via the Kubernetes API server, as detailed in [docs/architecture.md](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md).

The API Server (ateapi)

The ateapi binary serves as the entry point for control plane operations. It exposes a gRPC interface defined in [pkg/proto/ateapipb/ateapi.pb.go](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go).

gRPC Interface Definition

The protobuf definitions in ateapi.pb.go generate the Go structs and service interfaces used for all control plane communication. These definitions cover actor lifecycle operations including creation, suspension, resumption, and termination.

// Conceptual usage based on ateapi.pb.go generated types
client := ateapipb.NewActorManagementClient(conn)
resp, err := client.CreateActor(ctx, &ateapipb.CreateActorRequest{
    TemplateRef: "actor-template-001",
    WorkerPool:  "pool-us-east-1",
})

Server Implementation

Located in cmd/ateapi, the server initializes the gRPC listener and registers the service implementations. It translates incoming gRPC calls into Kubernetes API operations, creating or updating CRD instances that represent the desired actor state.

The Kubernetes Controller (atecontroller)

The atecontroller binary implements the reconciliation logic that bridges declarative CRD state with actual worker node capacity. It uses the controller-runtime framework to watch resources and trigger reconcile loops.

CRD Reconciliation Logic

The primary controllers reside in cmd/atecontroller/internal/controllers/:

• [workerpool_controller.go](https://github.com/agent-substrate/substrate/blob/main/cmd/atecontroller/internal/controllers/workerpool_controller.go) – Manages the lifecycle of worker pools, tracking capacity and scheduling decisions
• [actortemplate_controller.go](https://github.com/agent-substrate/substrate/blob/main/cmd/atecontroller/internal/controllers/actortemplate_controller.go) – Handles actor template versioning and validation

// Conceptual reconcile loop pattern as implemented in workerpool_controller.go
func (r *WorkerPoolReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var pool substratev1.WorkerPool
    if err := r.Get(ctx, req.NamespacedName, &pool); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }
    
    // Update pool status based on available atelet nodes
    pool.Status.AvailableCapacity = r.calculateCapacity(ctx, &pool)
    return ctrl.Result{RequeueAfter: 30 * time.Second}, r.Status().Update(ctx, &pool)
}

State Management via etcd

The controller persists all state through the Kubernetes API server, storing CRDs (Actor, ActorTemplate, WorkerPool) in etcd. This design leverages Kubernetes' distributed storage without requiring a separate database for the control plane.

The Node Agent (atelet)

Running on each worker node, the atelet binary receives gRPC commands from the atecontroller and manages the actual sandbox processes (ateom instances).

gRPC Command Execution

The atelet exposes a gRPC server that accepts lifecycle commands:

• StartActor – Mounts volume snapshots and launches the ateom sandbox
• SuspendActor – Checkpoints process state and uploads to object storage
• ResumeActor – Restores checkpoint and resumes execution

CSI Volume Integration

The atelet interacts with container storage through the CSI client implementation in [internal/volume/csi/client.go](https://github.com/agent-substrate/substrate/blob/main/internal/volume/csi/client.go):

// Volume mounting pattern from internal/volume/csi/client.go
func (c *CSIClient) MountSnapshot(ctx context.Context, snapshotID, targetPath string) error {
    req := &csi.NodePublishVolumeRequest{
        VolumeId:          snapshotID,
        TargetPath:        targetPath,
        VolumeCapability:  c.getVolumeCapability(),
        Readonly:          true,
    }
    _, err := c.nodeClient.NodePublishVolume(ctx, req)
    return err
}

Data Model and Protocol Buffers

The control plane uses a dual-purpose data model where Protocol Buffer definitions serve as both the wire format and the source of truth for CRD generation.

The generated Go code in [pkg/proto/ateapipb/ateapi.pb.go](https://github.com/agent-substrate/substrate/blob/main/pkg/proto/ateapipb/ateapi.pb.go) provides:

• Message structs for gRPC communication between ateapi, atecontroller, and atelet
• JSON tags for Kubernetes CRD serialization
• Validation methods for resource integrity

This approach ensures consistency between the API schema, database storage, and cross-service communication.

Observability and Testing

The control plane integrates OpenTelemetry for distributed tracing across all three components. The atecontroller uses controller-runtime's envtest package for integration testing against a local Kubernetes API server without requiring a full cluster, validating the reconcile logic in workerpool_controller.go against ephemeral etcd instances.

Summary

• Agent Substrate's control plane consists of three binaries: ateapi (API layer), atecontroller (reconciliation), and atelet (execution)
• State persistence relies on Kubernetes CRDs stored in etcd, accessed via the standard Kubernetes API
• Communication occurs over gRPC using Protocol Buffer definitions from pkg/proto/ateapipb/ateapi.pb.go
• Controller logic in workerpool_controller.go and actortemplate_controller.go manages resource scheduling using controller-runtime
• Node operations are handled by atelet, which interfaces with CSI storage via internal/volume/csi/client.go
• Design philosophy keeps the control plane focused and lightweight, delegating infrastructure management to Kubernetes while optimizing for high-frequency actor lifecycle operations as defined in [docs/glossary.md](https://github.com/agent-substrate/substrate/blob/main/docs/glossary.md)

Frequently Asked Questions

What is the role of the ateapi binary in Agent Substrate?

The ateapi binary serves as the gRPC API server and primary entry point for the control plane. It validates incoming requests for actor lifecycle management and translates them into Kubernetes CRD operations, persisting the desired state in etcd via the Kubernetes API server.

How does atecontroller interact with Kubernetes?

The atecontroller uses the controller-runtime library to establish watch streams on Custom Resource Definitions (ActorTemplate, Actor, WorkerPool). When resources change, it triggers reconcile loops in controllers like workerpool_controller.go to match the actual cluster state with the desired state defined in the CRDs.

What is the relationship between atelet and the control plane?

atelet acts as the node-local extension of the control plane. While ateapi and atecontroller manage global state and scheduling decisions, atelet executes the actual commands on worker nodes via gRPC, managing ateom sandbox processes and handling volume mounts through the CSI client.

Where is the control plane state stored?

All control plane state persists in etcd through the Kubernetes API server. The ateapi server creates CRD instances, atecontroller reads and updates their status, and atelet reports node-level metrics back to the controller, creating a unified source of truth for the entire system.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}