What Is the Purpose of the Atelet Component in Agent Substrate?

The atelet is a node-level supervisor DaemonSet that manages worker pod pools, delegates sandbox lifecycle operations to the ateom process, assembles OCI bundles, and streams snapshots to durable storage.

The atelet component serves as the critical bridge between the Kubernetes node and the Agent Substrate control plane. Operating as a DaemonSet in the ate-system namespace, it translates high-level actor scheduling decisions into concrete container operations while maintaining a minimal security footprint. Understanding the purpose of the atelet component is essential for debugging checkpoint/restore flows and optimizing resource utilization in the agent-substrate/substrate repository.

Core Responsibilities of the Atelet Component

Node-Level Supervision and Pod Herding

According to the architecture documentation, the atelet functions as a node supervisor that maintains a pool of ready worker pods on each node docs/architecture.md#L13-L19. It "herds" these physical containers by communicating with the ate-api-server to obtain work assignments for incoming actors.

This architecture decouples the lifecycle of the physical Kubernetes pod from the sandboxed workload running inside it. When the control plane schedules an actor, the atelet selects an available worker pod rather than creating a new one, significantly reducing cold-start latency and resource churn.

Delegating Sandbox Operations to Ateom

Crucially, the atelet does not execute sandboxed workloads directly. Instead, it forwards lifecycle commands—such as run, checkpoint, and restore—to an ateom process that resides inside each worker pod docs/architecture.md#L19-L22.

This delegation model maintains a strict security boundary. The DaemonSet itself requires no additional Linux capabilities, as all privileged container operations occur inside the isolated worker pod environment managed by ateom.

OCI Bundle Preparation

The atelet handles the "pull-and-assemble" phase of container initialization. It pulls container images from registries, constructs valid OCI bundles, and unpacks them on the node's filesystem before the workload starts docs/glossary.md#L57-L60.

This preparation ensures that worker pods receive fully resolved filesystem layers, allowing ateom to execute container runtime commands without image-pull overhead during critical path operations.

Snapshot Streaming and Storage Movement

When an actor is suspended or paused, the atelet coordinates snapshot persistence. It streams checkpoint data generated by ateom to durable storage backends such as Google Cloud Storage or Amazon S3 docs/architecture.md#L22-L24.

During resume operations, the process reverses: the atelet downloads the snapshot from storage and hands it to the local ateom process for restoration. This storage-mover capability enables migration of actor state across nodes and long-term hibernation of inactive workloads.

Operational Guide: Interacting with Atelet

Verify the DaemonSet Deployment

To confirm that the atelet is running on all eligible nodes, inspect the DaemonSet and its pod inventory:

# Check DaemonSet rollout status

kubectl -n ate-system get daemonset atelet

# List atelet pods across the cluster

kubectl -n ate-system get pods -l app=atelet -o wide

The complete DaemonSet specification, including update strategies and volume mounts, is defined in manifests/ate-install/atelet.yaml.

Inspecting Logs for Snapshot Operations

When debugging checkpoint failures or slow restores, examine the atelet logs for storage transfer details:

# Capture logs from a specific node

POD=$(kubectl -n ate-system get pods -l app=atelet -o jsonpath='{.items[0].metadata.name}')
kubectl -n ate-system logs -f $POD --tail=500

Log entries typically indicate snapshot upload completion, download initiation, and any communication timeouts with the local ateom processes.

Programmatic Access via gRPC

The atelet exposes a gRPC API defined in internal/proto/ateletpb/atelet.proto. The following Go client demonstrates listing active workers on a node:

package main

import (
    "context"
    "log"
    "google.golang.org/grpc"
    pb "github.com/agent-substrate/substrate/internal/proto/ateletpb"
)

func main() {
    // Connect to local atelet service (default port 8085)
    conn, err := grpc.Dial("localhost:8085", grpc.WithInsecure())
    if err != nil {
        log.Fatalf("connection failed: %v", err)
    }
    defer conn.Close()

    client := pb.NewAteletClient(conn)
    
    // Request current worker inventory
    resp, err := client.ListWorkers(context.Background(), &pb.ListWorkersRequest{})
    if err != nil {
        log.Fatalf("ListWorkers failed: %v", err)
    }
    
    log.Printf("Active workers on node: %v", resp.Workers)
}

Triggering Manual Checkpoints

Use the kubectl-ate CLI to initiate a checkpoint, which causes the control plane to instruct the atelet to capture and upload a snapshot:

# Suspend an actor named "analytics" in the "production" atespace

kubectl ate suspend actor production/analytics

This command validates the end-to-end flow from control plane to atelet to ateom.

Observability Configuration

The atelet exposes standard Kubernetes health probes and Prometheus metrics. The DaemonSet configuration defines readiness (/readyz) and liveness (/healthz) endpoints, along with a metrics port (/metrics) for monitoring checkpoint latency and storage transfer rates manifests/ate-install/atelet.yaml#L71-L88.

Summary

• The atelet runs as a DaemonSet on every node, acting as the primary node supervisor for the Agent Substrate system.
• It herds pools of worker pods and delegates actual sandbox execution to the ateom process inside those pods, maintaining a minimal privilege model.
• The component handles OCI bundle assembly, pulling and unpacking container images before workload assignment.
• It manages bidirectional snapshot streaming between local worker pods and durable cloud storage (GCS/S3).
• All operations are accessible via a gRPC interface defined in atelet.proto, with operational visibility through standard Kubernetes logs and Prometheus metrics.

Frequently Asked Questions

How does the atelet differ from the ateom process?

While the atelet is a node-level DaemonSet that manages scheduling and storage, ateom is a process running inside each worker pod that executes the actual container runtime commands. The atelet sends gRPC commands to ateom to trigger checkpoints and restores, maintaining a security boundary where the node agent remains unprivileged docs/architecture.md#L19-L22.

Why is the atelet deployed as a DaemonSet rather than a Deployment?

A DaemonSet ensures exactly one atelet pod runs on every node that can host actors. This provides local access to the node's filesystem for OCI bundle storage and enables direct communication with the kubelet and container runtime. A Deployment would not guarantee the node coverage required for local snapshot management and immediate work assignment.

What happens if the atelet pod restarts during an active checkpoint?

The atelet is designed to be stateless regarding ongoing operations. If it restarts, it reconnects to existing worker pods and resumes tracking their status. Incomplete snapshot uploads will timeout and retry based on control plane coordination, ensuring no orphaned processes remain on the node. Check the logs for snapshot resumption messages to verify recovery.

Which ports does the atelet expose for communication?

According to the manifest, the atelet exposes port 8085 for gRPC API traffic from the control plane, port 9090 for Prometheus metrics, and standard HTTP ports for Kubernetes health probes manifests/ate-install/atelet.yaml#L71-L88.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}