# What Is Agent Substrate? A Deep Dive into the High-Density Actor Runtime > Discover Agent Substrate, a high-density runtime enabling millions of concurrent agents with sub-second latency. Learn how it maps workloads efficiently onto worker pools for optimal performance. - Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate) - Tags: deep-dive - Published: 2026-08-22 --- **Agent Substrate is a high-density, low-latency runtime that maps a large set of agent-like workloads (actors) onto a smaller pool of pre-started workers, enabling millions of concurrent agents with sub-second activation latency.** Agent Substrate is an open-source project (`agent-substrate/substrate`) designed for large-scale AI agent deployments. According to the source code and architecture documentation, it delivers a performant runtime environment by decoupling high-frequency actor state management from the Kubernetes API, achieving heavy multiplexing of workloads on minimal infrastructure. ## Core Architecture and Components The system extends Kubernetes with a specialized control plane that handles the fast suspend and resume lifecycle of actors. This architecture consists of three primary layers working in concert. ### Control Plane (ate-api-server) The brain of the operation resides in [`cmd/ateapi/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/main.go), implementing the `ate-api-server`. This component stores actor-to-worker mappings in a high-performance Redis store, schedules workers, and orchestrates snapshot restoration and checkout. By maintaining this state outside the standard Kubernetes etcd, the control plane achieves the throughput necessary for millions of concurrent **actors**. ### Node Supervisor (atelet and ateom) Running on each node as a DaemonSet, the Node Supervisor—implemented across [`cmd/atelet/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atelet/main.go) and the `ateom` component—manages the physical **worker** pods. This layer streams **snapshots** to durable storage and invokes sandbox runtimes such as gVisor or micro-VMs to isolate actor execution. It acts as the local agent that translates control plane decisions into container operations. ### Networking Stack (atenet) Located in [`cmd/atenet/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/main.go), the networking layer provides actor-aware DNS and Envoy routing. The **atenet** controller can pause an incoming request, trigger a resume operation for a suspended actor, and then transparently forward traffic to the appropriate worker. This enables seamless access to agents regardless of their current active or suspended state. ## Key Concepts and Resource Model Agent Substrate introduces a specific vocabulary to manage its unique workload patterns: - **Actor**: An instance of an agent-like workload (e.g., a LangChain agent) that represents the unit of computation. - **WorkerPool**: A pool of "warm" pods maintained by the system, ready to host resumed actors without cold-start penalties. - **ActorTemplate**: An immutable definition specifying the actor's container image, configuration, and resource limits. - **Snapshot**: Persistent memory and filesystem state stored in GCS or other durable storage, capturing the exact execution state of an actor. - **Suspend/Resume**: The core lifecycle operations where actors are checkpointed when idle and restored on-demand, typically achieving **< 100ms** resume latency for the 95th percentile of requests. ## The Actor Lifecycle and Performance The runtime decouples actor existence from resource consumption. When an actor becomes idle, the system initiates a **suspend** operation, checkpointing state to external storage and freeing the worker for other tasks. Upon receiving a new request, the **atenet** router triggers a **resume** operation, restoring the snapshot to an available worker pod. This model targets aggressive performance metrics: supporting millions of concurrent actors while maintaining activation latency under one second. The architecture documentation in [`docs/architecture.md`](https://github.com/agent-substrate/substrate/blob/main/docs/architecture.md) explicitly defines this as the "North Star" metric, driving design decisions around Redis-based state storage and efficient snapshot serialization. ## Managing Substrate with kubectl-ate Interactions with the system occur through `kubectl-ate`, a CLI tool implemented in [`cmd/kubectl-ate/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/kubectl-ate/main.go). The following examples demonstrate the typical operational workflow. ### 1. Create an Atespace An **Atespace** serves as a logical namespace for grouping actors: ```bash kubectl ate create atespace demo ``` This command creates the `Atespace` custom resource, providing isolation for the subsequent actor deployments. ### 2. Deploy an Actor Instantiate an actor from an existing template: ```bash kubectl ate create actor my-counter-1 -a demo \ --template=ate-demo-counter/counter ``` This creates an `Actor` resource linked to the `counter` template, initially in a suspended state with no allocated worker. ### 3. Resume an Actor While resumes typically trigger automatically upon request arrival, you can force a manual activation: ```bash kubectl ate resume actor my-counter-1 -a demo ``` This instructs the control plane to assign a worker from the pool and restore the actor's snapshot. ### 4. Send a Request Access the resumed actor through the intelligent routing layer: ```bash curl -X POST -H "Host: my-counter-1.demo.actors.resources.substrate.ate.dev" \ http://localhost:8000/ ``` The `atenet` router resolves the DNS name, handles any necessary resume operations, and proxies the HTTP request to the active worker. ### 5. Suspend an Idle Actor Explicitly checkpoint and free resources: ```bash kubectl ate suspend actor my-counter-1 -a demo ``` This validates the actor's state, commits the latest snapshot to GCS, and returns the underlying worker to the available pool. ## Summary Agent Substrate redefines large-scale agent deployment by introducing a specialized orchestration layer atop Kubernetes: - **Multiplexing Architecture**: Maps many actors to few workers through aggressive snapshotting and state decoupling. - **Sub-100ms Resume**: Targets millisecond-scale activation latency for suspended agents. - **Three-Tier Design**: Separates concerns between the control plane ([`cmd/ateapi/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateapi/main.go)), node supervision ([`cmd/atelet/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atelet/main.go)), and networking ([`cmd/atenet/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/main.go)). - **Kubernetes-Native**: Leverages existing infrastructure for pod management while bypassing etcd for high-frequency state changes. - **CLI-First Operations**: Provides complete lifecycle management through the `kubectl-ate` command-line interface. ## Frequently Asked Questions ### How does Agent Substrate differ from standard Kubernetes? Standard Kubernetes schedules pods as long-running processes, creating a 1:1 relationship between workload instances and resource consumption. Agent Substrate introduces a **suspend/resume** mechanism that decouples the logical actor from the physical worker pod, allowing many actors to time-share a smaller worker pool. This achieves higher density and faster scaling than native Kubernetes Deployments or Knative serverless containers. ### What is the typical cold-start latency for an actor? According to the architecture documentation, the system targets a **95th-percentile resume latency of under 100 milliseconds**. This assumes the snapshot data resides in fast storage (such as GCS with appropriate caching) and a warm WorkerPool is available. Cold starts from complete suspension rarely exceed one second in properly configured clusters. ### How does the networking layer handle suspended actors? The **atenet** component, defined in [`cmd/atenet/main.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/main.go), implements actor-aware DNS and Envoy-based request buffering. When traffic arrives for a suspended actor, the networking stack pauses the request, signals the control plane to resume the target actor, then forwards the traffic once the snapshot restores to a worker. This process is transparent to the client application. ### Where is actor state persisted? Actor state persists as **snapshots** in durable object storage such as Google Cloud Storage (GCS). The `atelet` DaemonSet streams memory and filesystem state from the worker pods to this backend during suspend operations. Upon resume, the control plane selects an available worker and orchestrates the checkout and restoration of this snapshot data before marking the actor as active.