Where Is Dynamic Instance State Stored in Agent Substrate?

Dynamic instance state in Agent Substrate is persisted in the InstanceStatus sub-resource of a Kubernetes Custom Resource backed by etcd, while each atelet worker maintains a local in-memory cache via informers for low-latency access.

Agent Substrate treats every logical workload as an instance (also called an actor). The mutable, runtime-specific information for each instance—such as its current phase, health metrics, assigned worker pod, and per-instance configuration—is stored in a Custom Resource (CR) that the control plane persists to etcd via the Kubernetes API. This architecture separates immutable desired state from dynamic observed state, ensuring durability across control plane and worker restarts.

The Instance Custom Resource Definition

The schema that dictates where dynamic state lives is declared in pkg/api/v1alpha1/instance_types.go. This file defines the Instance type, which contains two primary top-level fields: Spec (desired state) and Status (observed, dynamic state).

InstanceSpec vs. InstanceStatus

• InstanceSpec – Contains largely immutable configuration such as container images, resource limits, and environment variables defined at creation time.
• InstanceStatus – Houses mutable fields that change during execution, including Phase, AssignedWorker, LastSeen, and Metrics. These fields constitute the dynamic instance state that the system tracks at runtime.

Persistence Layer: etcd via the Kubernetes API

The authoritative source of dynamic instance state resides in the Kubernetes API server’s etcd store. The controller implementation in internal/instance/controller.go reconciles desired and observed state by reading and writing Instance objects through the generated client in pkg/client/clientset.go.

When an atelet worker updates runtime details—such as assigning itself to an instance—it invokes the client to patch the InstanceStatus sub-resource. The API server immediately persists this change to etcd, ensuring that dynamic state survives process restarts and can be reconstructed by newly elected controllers.

Local Caching in Atelet Workers

To prevent excessive load on the API server, each atelet (worker) component runs an informer that watches Instance resources. The local cache implementation in internal/atelet/instance_cache.go stores a read-only snapshot of InstanceStatus fields in memory. The worker initialization logic in cmd/atelet/main.go sets up these informers before the worker begins processing requests.

This cache provides O(1) lookup latency for the current state but is eventually consistent; all mutations must still flow through the Kubernetes API to ensure etcd remains the single source of truth.

Working with Dynamic Instance State in Code

The following examples demonstrate how to interact with dynamic state using the Substrate client.

Creating an Instance

import (
    substratev1alpha1 "github.com/agent-substrate/substrate/pkg/api/v1alpha1"
    metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
    clientset "github.com/agent-substrate/substrate/pkg/client"
)

func createInstance(cs *clientset.Clientset, name, ns string) error {
    inst := &substratev1alpha1.Instance{
        ObjectMeta: metav1.ObjectMeta{
            Name:      name,
            Namespace: ns,
        },
        Spec: substratev1alpha1.InstanceSpec{
            Image: "my/app:latest",
        },
    }
    _, err := cs.SubstrateV1alpha1().Instances(ns).Create(context.Background(), inst, metav1.CreateOptions{})
    return err
}

Reading Dynamic State

func getInstanceStatus(cs *clientset.Clientset, name, ns string) (*substratev1alpha1.InstanceStatus, error) {
    inst, err := cs.SubstrateV1alpha1().Instances(ns).Get(context.Background(), name, metav1.GetOptions{})
    if err != nil {
        return nil, err
    }
    return &inst.Status, nil
}

Updating Runtime State

func assignWorker(cs *clientset.Clientset, name, ns, worker string) error {
    inst, err := cs.SubstrateV1alpha1().Instances(ns).Get(context.Background(), name, metav1.GetOptions{})
    if err != nil {
        return err
    }

    inst.Status.AssignedWorker = worker
    inst.Status.Phase = substratev1alpha1.InstanceRunning

    _, err = cs.SubstrateV1alpha1().Instances(ns).UpdateStatus(context.Background(), inst, metav1.UpdateOptions{})
    return err
}

These snippets illustrate that mutable runtime data lives inside the InstanceStatus sub-resource, which is persisted by the Kubernetes API server and cached locally by atelet workers.

Summary

• Authoritative Storage: Dynamic instance state is stored in the Instance Custom Resource's Status field, persisted to etcd by the Kubernetes API server.
• Controller Logic: The file internal/instance/controller.go manages reconciliation and writes to the API server, using the client defined in pkg/client/clientset.go.
• Worker Caching: Atelet workers use an in-memory cache (internal/atelet/instance_cache.go) populated by informers initialized in cmd/atelet/main.go for fast, local read access.
• Durability: Because state resides in etcd, instance runtime information survives control plane and worker restarts without data loss.

Frequently Asked Questions

Is dynamic instance state stored in etcd or only in memory?

The authoritative copy is stored in etcd via the Kubernetes API. Workers maintain an in-memory cache for performance, but all mutations flow through the API server to ensure persistence across restarts.

What is the difference between InstanceSpec and InstanceStatus?

InstanceSpec defines the desired, largely immutable configuration (image, resources), while InstanceStatus holds the dynamic, runtime-specific fields such as Phase, AssignedWorker, and health metrics that change as the instance executes.

How do workers access instance state without overwhelming the API server?

Each atelet worker runs a Kubernetes informer that watches Instance resources and populates a local cache defined in internal/atelet/instance_cache.go. This allows O(1) local lookups instead of querying the API server for every read.

Can instance state persist if the control plane restarts?

Yes. Because the state is stored in etcd via the Custom Resource, the control plane can reconstruct the full runtime view of all instances after a restart by reading the Instance objects from the API server.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}