# How the Ateom Sidecar Works with Sandbox Runtimes in Agent Substrate

> Learn how the ateom sidecar orchestrates gVisor containers and micro-VMs in Agent Substrate worker pods abstracting runtime differences and managing image caching resource isolation and coordination.

- Repository: [Agent Substrate/substrate](https://github.com/agent-substrate/substrate)
- Tags: how-to-guide
- Published: 2026-08-22

---

**The ateom sidecar orchestrates the complete lifecycle of gVisor containers and micro-VMs inside Agent Substrate worker pods, abstracting runtime differences while handling image caching, resource isolation, and coordination with companion sidecars.**

Agent Substrate is an open-source framework for running isolated agents (actors) in Kubernetes. The **ateom sidecar** acts as the per-worker runtime manager that translates `SandboxConfig` custom resources into live sandboxed processes, enforcing security boundaries defined in the project's threat model.

## Core Responsibilities and Architecture

The sidecar operates as a privileged init container within each worker pod. It exposes a Unix-domain socket for control-plane communication and maintains the state of all sandboxes on that node.

Key duties include:

- **Sandbox Lifecycle Management**: Watches the `SandboxConfig` CRD (defined in [`pkg/api/v1alpha1/sandboxconfig_types.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/api/v1alpha1/sandboxconfig_types.go)) to create and destroy sandboxes.
- **Runtime Abstraction**: Implements a unified interface over divergent backends—gVisor containers and micro-VMs.
- **Image Distribution**: Pulls actor images via containerd/cri APIs and caches layers locally.
- **Resource Enforcement**: Applies CPU and memory limits from the `ActorTemplate` spec.
- **Telemetry Export**: Collects per-sandbox statistics via ttrpc and forwards them to the central controller.

## Runtime Abstraction: gVisor vs. Micro-VMs

The sidecar determines the isolation backend by inspecting the `sandboxType` field in the `SandboxConfig` and delegates to the appropriate launcher in [`internal/ateompath/ateompath.go`](https://github.com/agent-substrate/substrate/blob/main/internal/ateompath/ateompath.go).

### gVisor Container Backend

For `gvisor` types, the sidecar invokes the `runsc` binary directly. It configures user namespaces, seccomp filters, and gVisor's sentry to provide kernel-masking isolation without hardware virtualization.

### Micro-VM Backend

For `microvm` types, the sidecar spawns the `ateom-microvm` binary. The implementation in [`cmd/ateom-microvm/run_test.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateom-microvm/run_test.go) demonstrates how the sidecar prepares the root filesystem, sets up the micro-VM process, and establishes communication channels. Both backends expose identical `ActorContainer` metadata to the rest of the system.

```go
// Conceptual flow based on internal/ateompath/ateompath.go
func StartSandbox(cfg *SandboxConfig) error {
    if cfg.Spec.Type == "gvisor" {
        return runscStart(cfg.Spec.Image, cfg.Spec.Resources)
    }
    if cfg.Spec.Type == "microvm" {
        return microvmStart(cfg) // Delegates to ateom-microvm
    }
    return fmt.Errorf("unknown sandbox type %q", cfg.Spec.Type)
}

```

## Image Caching and Filesystem Setup

Before launching any sandbox, the sidecard pulls the required actor image and caches it in `/var/lib/ateom` on the host. This ensures that subsequent actor restarts or scale-out events do not trigger redundant registry pulls. The root filesystem is then mounted into the sandbox namespace, read-only where possible, to prevent mutation of the base image.

## Security Boundaries and Privilege Model

The sidecar runs under a restricted ServiceAccount with a limited Capability set, as documented in [`docs/threat-model.md`](https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md). This design ensures that even if an attacker escapes the sandboxed actor, they gain only the minimal privileges of the sidecar process, not full host access.

Isolation mechanisms include:

- **User Namespaces**: Separate UID/GID mappings for each sandbox.
- **Seccomp Profiles**: System call filtering applied by gVisor or the micro-VM monitor.
- **Resource Limits**: Enforced via cgroups based on the `ActorTemplate` validation logic (see [`actortemplate-validation-test.go`](https://github.com/agent-substrate/substrate/blob/main/actortemplate-validation-test.go)).

## Telemetry Collection and Sidecar Coordination

### Metrics Export

While a sandbox is running, the sidecar collects CPU, memory, and I/O statistics through the ttrpc service defined in [`cmd/ateom-microvm/stats.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateom-microvm/stats.go). These metrics are forwarded to the controller using the gRPC interface specified in `internal/proto/ateompb/ateom.proto`.

```go
// Derived from cmd/ateom-microvm/stats.go
srv := newStatsService(agentID, "app_overlay", "sidecar_overlay")
if err := srv.Start(); err != nil {
    return err
}
defer srv.Stop()

snapshot := srv.Snapshot() // CPU, memory, I/O counters

```

### Graceful Shutdown Coordination

The sidecar acts as a coordination point for other containers in the pod, such as the Envoy sidecar for networking. When a worker drains, it signals companion sidecars via the logic in [`cmd/atenet/internal/router/envoydrain.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/internal/router/envoydrain.go) to ensure zero-downtime transitions.

```go
// Pattern from cmd/atenet/internal/router/envoydrain.go
drainer := envoyDrainer{
    adminAddr: cfg.EnvoyAdminAddr,
}
if err := drainer.Drain(ctx); err != nil {
    log.Printf("envoy drain failed: %v", err)
}

```

## Summary

- The **ateom sidecar** watches `SandboxConfig` CRDs in [`pkg/api/v1alpha1/sandboxconfig_types.go`](https://github.com/agent-substrate/substrate/blob/main/pkg/api/v1alpha1/sandboxconfig_types.go) to trigger sandbox creation and teardown.
- It abstracts **gVisor** and **micro-VM** runtimes behind a common interface implemented in [`internal/ateompath/ateompath.go`](https://github.com/agent-substrate/substrate/blob/main/internal/ateompath/ateompath.go).
- Images are cached in `/var/lib/ateom` to reduce startup latency and registry load.
- Resource constraints and security policies are enforced via user namespaces, seccomp, and limited Capabilities per [`docs/threat-model.md`](https://github.com/agent-substrate/substrate/blob/main/docs/threat-model.md).
- Runtime metrics flow through the ttrpc service in [`cmd/ateom-microvm/stats.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateom-microvm/stats.go) to the controller.
- Coordination with companion sidecars uses Unix-domain sockets and graceful drain handlers like those in [`cmd/atenet/internal/router/envoydrain.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/atenet/internal/router/envoydrain.go).

## Frequently Asked Questions

### What is the ateom sidecar in Agent Substrate?

The ateom sidecar is a per-worker component that manages isolated sandbox runtimes for agents. It runs inside every worker pod, translates `SandboxConfig` resources into running gVisor containers or micro-VMs, and handles image pulling, resource enforcement, and metrics export.

### How does ateom choose between gVisor and micro-VMs?

The sidecar inspects the `sandboxType` field in the `SandboxConfig` spec. If the value is `gvisor`, it launches a gVisor container via `runsc`; if `microvm`, it delegates to the `ateom-microvm` binary, as orchestrated by the logic in [`internal/ateompath/ateompath.go`](https://github.com/agent-substrate/substrate/blob/main/internal/ateompath/ateompath.go).

### Where does ateom store pulled container images?

The sidecar caches images in `/var/lib/ateom` on the worker node using standard containerd/cri APIs. This local cache allows multiple actors sharing the same base image to start without redundant network transfers.

### How does the sidecar report sandbox metrics to the controller?

It exposes a ttrpc stats service defined in [`cmd/ateom-microvm/stats.go`](https://github.com/agent-substrate/substrate/blob/main/cmd/ateom-microvm/stats.go) that samples CPU, memory, and I/O counters from each sandbox. These statistics are serialized and sent to the central controller via the gRPC protocol defined in `internal/proto/ateompb/ateom.proto`.