# How Cubelet Manages the Local Sandbox Lifecycle in CubeSandbox

> Discover how Cubelet orchestrates the local sandbox lifecycle, managing creation, execution, pausing, resuming, and cleanup with local storage pools, Redis state tracking, and lifecycle plugins.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-16

---

**Cubelet orchestrates the complete sandbox lifecycle—from creation and execution to pause, resume, and cleanup—through a coordinated system of local storage pools, Redis-backed state tracking, and lifecycle manager plugins.**

In the **TencentCloud/CubeSandbox** architecture, Cubelet runs as the core agent inside each guest VM, handling all heavyweight storage operations while synchronizing state with the central lifecycle manager via Redis. This article examines the actual source code implementation to explain how Cubelet deterministically manages sandbox lifecycles with fault tolerance and exclusive state ownership.

## Sandbox Metadata and State Tracking

Every sandbox begins with persistent metadata registration and transient state coordination between Cubelet and the lifecycle manager.

### Registry and Metadata Schema

The **registry** ([`cube-lifecycle-manager/internal/registry/registry.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/registry/registry.go)) stores a `SandboxLifecycleMeta` record for each sandbox. This struct contains critical flags including `AutoPause` and `TimeoutSeconds` that determine how the lifecycle manager orchestrates the sandbox.

```go
meta := lifecycle.SandboxLifecycleMeta{
    SandboxID:      sandboxID,
    AutoPause:      true,
    TimeoutSeconds: lifecycle.TimeoutSecondsPtr(300),
}
registry.Upsert(meta)

```

### Redis State Keys and Exclusive Ownership

The **Redis schema** ([`cube-lifecycle-manager/internal/redisstream/stream.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/redisstream/stream.go)) maintains two keys per sandbox:

- **MetaKey**: The persistent metadata record
- **StateKey**: A transient flag indicating the current phase (running, paused, stopped)

Cubelet acquires exclusive ownership using atomic Redis operations:

```go
if err := client.AcquireState(ctx, sandboxID); err != nil {
    // another manager already owns the sandbox
}

```

The `SET NX EX` operation guarantees that only one lifecycle manager component can control the sandbox at any moment, preventing split-brain scenarios during distributed coordination.

## Creation and Volume Provisioning

When a sandbox creation request arrives, Cubelet initializes the local storage infrastructure before launching the container runtime.

### Storage Pool Initialization

Cubelet creates a **local storage pool** using implementations in [`Cubelet/storage/pool.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool.go) and [`Cubelet/storage/pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool_withreflink.go). These pools manage the underlying filesystem resources that back the sandbox volumes.

### Volume Allocation

The **volume** representing the sandbox filesystem is provisioned through [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go). This volume is attached to the container runtime, and the sandbox process launches with the allocated storage:

```go
pool, _ := storage.NewPool(...)
vol, err := pool.AllocVolume(sandboxID)
if err != nil { return err }

```

This local-first approach keeps I/O-intensive operations within the VM boundaries while the lifecycle manager handles only lightweight metadata coordination.

## Execution and Monitoring

Once running, Cubelet maintains the sandbox's active state through continuous monitoring and heartbeat mechanisms.

### Workflow Plugin Monitoring

The **workflow plugin** (`Cubelet/plugins/workflow`) monitors the sandbox container process, tracking health and execution status. This plugin runs as part of Cubelet's internal process management system.

### Heartbeat and Activity Tracking

Periodic heartbeats update the sandbox's **last-active timestamp** in the registry. The sweeper component uses this timestamp to calculate idle time and determine whether to trigger auto-pause or reclamation policies.

## Auto-Pause and Timeout Handling

The **sweeper** ([`cube-lifecycle-manager/internal/sweeper/sweeper.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/sweeper/sweeper.go)) runs on the master side and implements the idle timeout logic. It reads all sandbox metadata via the `Bootstrap` method and evaluates each sandbox against its `AutoPause` and `TimeoutSeconds` configuration.

When a sandbox exceeds its idle threshold, the sweeper either:

1. Sends a **pause** command to Cubelet, which writes the *paused* state key to Redis
2. Triggers **reclamation** for permanent cleanup, including volume deletion and snapshot cleanup

```go
if meta.AutoPause && idleTooLong {
    client.Pause(ctx, sandboxID)   // writes paused state key
}

```

This architecture separates policy decisions (sweeper) from execution (Cubelet), allowing the master to manage thousands of sandboxes while Cubelet handles the actual storage operations.

## Resume and Wake-Up

Paused sandboxes resume through the **resumer** ([`cube-lifecycle-manager/internal/resumer/resumer.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/resumer/resumer.go)), which watches the state key for wake-up signals.

When a paused sandbox receives new activity, the resumer:

1. Clears the pause flag in Redis
2. Signals Cubelet to re-attach the volume
3. Restores the sandbox to running state

```go
if client.HasPendingWork(sandboxID) {
    client.Resume(ctx, sandboxID)  // clears paused flag, re‑attaches volume
}

```

The state transition maintains data consistency by ensuring the volume is fully re-attached before the sandbox process resumes execution.

## Cleanup and Resource Reclamation

When a sandbox terminates—whether through normal completion, timeout, or explicit deletion—Cubelet executes a deterministic cleanup sequence.

### Template Cleanup and Volume Removal

Cubelet invokes the **template cleanup** routines in [`Cubelet/storage/template_cleanup.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/template_cleanup.go) to remove all associated resources:

- Sandbox volumes allocated in [`local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/local.go)
- Filesystem snapshots
- Host-directory bindings and mount points

This cleanup ensures that storage pools remain unfragmented and that no orphaned resources persist in the guest VM after sandbox termination.

## Summary

- **Cubelet** manages the complete sandbox lifecycle inside each guest VM, from volume creation in [`Cubelet/storage/pool.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool.go) to final cleanup in [`template_cleanup.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/template_cleanup.go).
- **State coordination** uses Redis keys (`MetaKey` and `StateKey`) with atomic `SET NX EX` operations to guarantee exclusive ownership between Cubelet and the lifecycle manager.
- **Auto-pause functionality** is implemented by the sweeper ([`cube-lifecycle-manager/internal/sweeper/sweeper.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/sweeper/sweeper.go)) based on `TimeoutSeconds` and `AutoPause` flags stored in the registry.
- **Resume operations** are handled by the resumer ([`cube-lifecycle-manager/internal/resumer/resumer.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/resumer/resumer.go)), which clears pause states and re-attaches volumes.
- **Storage operations** remain local to the VM through [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go), while the master-side components manage only lightweight metadata and state transitions.

## Frequently Asked Questions

### What happens if Cubelet crashes during a sandbox pause operation?

If Cubelet crashes during a pause, the Redis state key remains in an inconsistent state until the lifecycle manager detects the failure. Because the sweeper uses the `Bootstrap` method to read all sandbox metadata on startup, it can identify sandboxes with stale state keys and either resume them or trigger reclamation based on the last recorded heartbeat timestamp. The `SandboxLifecycleMeta` persists in the registry regardless of Cubelet's availability.

### How does Cubelet ensure exclusive control of a sandbox?

Cubelet acquires exclusive control through the **AcquireState** method in [`cube-lifecycle-manager/internal/redisstream/stream.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/redisstream/stream.go), which executes a Redis `SET` command with `NX` (only if not exists) and `EX` (expiration) flags. This atomic operation prevents multiple managers from simultaneously controlling the same sandbox, ensuring deterministic state transitions during pause, resume, and delete operations.

### Can the auto-pause timeout be configured per sandbox?

Yes, each sandbox stores its own `TimeoutSeconds` value in the `SandboxLifecycleMeta` struct defined in [`cube-lifecycle-manager/internal/registry/registry.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/registry/registry.go). The sweeper evaluates this field individually for each sandbox when determining idle timeouts, allowing different sandboxes to have different auto-pause policies within the same Cubelet instance.

### What is the difference between the sweeper and resumer components?

The **sweeper** ([`cube-lifecycle-manager/internal/sweeper/sweeper.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/sweeper/sweeper.go)) proactively monitors sandbox activity and initiates pause sequences when idle timeouts occur, while the **resumer** ([`cube-lifecycle-manager/internal/resumer/resumer.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube-lifecycle-manager/internal/resumer/resumer.go)) reacts to incoming work signals by clearing pause flags and re-attaching volumes. The sweeper handles the "sleep" transition, and the resumer handles the "wake" transition, creating a bidirectional lifecycle management system.