How Cubelet Manages the Local Sandbox Lifecycle in CubeSandbox

Cubelet orchestrates the complete sandbox lifecycle—from creation and execution to pause, resume, and cleanup—through a coordinated system of local storage pools, Redis-backed state tracking, and lifecycle manager plugins.

In the TencentCloud/CubeSandbox architecture, Cubelet runs as the core agent inside each guest VM, handling all heavyweight storage operations while synchronizing state with the central lifecycle manager via Redis. This article examines the actual source code implementation to explain how Cubelet deterministically manages sandbox lifecycles with fault tolerance and exclusive state ownership.

Sandbox Metadata and State Tracking

Every sandbox begins with persistent metadata registration and transient state coordination between Cubelet and the lifecycle manager.

Registry and Metadata Schema

The registry (cube-lifecycle-manager/internal/registry/registry.go) stores a SandboxLifecycleMeta record for each sandbox. This struct contains critical flags including AutoPause and TimeoutSeconds that determine how the lifecycle manager orchestrates the sandbox.

meta := lifecycle.SandboxLifecycleMeta{
    SandboxID:      sandboxID,
    AutoPause:      true,
    TimeoutSeconds: lifecycle.TimeoutSecondsPtr(300),
}
registry.Upsert(meta)

Redis State Keys and Exclusive Ownership

The Redis schema (cube-lifecycle-manager/internal/redisstream/stream.go) maintains two keys per sandbox:

• MetaKey: The persistent metadata record
• StateKey: A transient flag indicating the current phase (running, paused, stopped)

Cubelet acquires exclusive ownership using atomic Redis operations:

if err := client.AcquireState(ctx, sandboxID); err != nil {
    // another manager already owns the sandbox
}

The SET NX EX operation guarantees that only one lifecycle manager component can control the sandbox at any moment, preventing split-brain scenarios during distributed coordination.

Creation and Volume Provisioning

When a sandbox creation request arrives, Cubelet initializes the local storage infrastructure before launching the container runtime.

Storage Pool Initialization

Cubelet creates a local storage pool using implementations in Cubelet/storage/pool.go and Cubelet/storage/pool_withreflink.go. These pools manage the underlying filesystem resources that back the sandbox volumes.

Volume Allocation

The volume representing the sandbox filesystem is provisioned through Cubelet/storage/local.go. This volume is attached to the container runtime, and the sandbox process launches with the allocated storage:

pool, _ := storage.NewPool(...)
vol, err := pool.AllocVolume(sandboxID)
if err != nil { return err }

This local-first approach keeps I/O-intensive operations within the VM boundaries while the lifecycle manager handles only lightweight metadata coordination.

Execution and Monitoring

Once running, Cubelet maintains the sandbox's active state through continuous monitoring and heartbeat mechanisms.

Workflow Plugin Monitoring

The workflow plugin (Cubelet/plugins/workflow) monitors the sandbox container process, tracking health and execution status. This plugin runs as part of Cubelet's internal process management system.

Heartbeat and Activity Tracking

Periodic heartbeats update the sandbox's last-active timestamp in the registry. The sweeper component uses this timestamp to calculate idle time and determine whether to trigger auto-pause or reclamation policies.

Auto-Pause and Timeout Handling

The sweeper (cube-lifecycle-manager/internal/sweeper/sweeper.go) runs on the master side and implements the idle timeout logic. It reads all sandbox metadata via the Bootstrap method and evaluates each sandbox against its AutoPause and TimeoutSeconds configuration.

When a sandbox exceeds its idle threshold, the sweeper either:

1. Sends a pause command to Cubelet, which writes the paused state key to Redis
2. Triggers reclamation for permanent cleanup, including volume deletion and snapshot cleanup

if meta.AutoPause && idleTooLong {
    client.Pause(ctx, sandboxID)   // writes paused state key
}

This architecture separates policy decisions (sweeper) from execution (Cubelet), allowing the master to manage thousands of sandboxes while Cubelet handles the actual storage operations.

Resume and Wake-Up

Paused sandboxes resume through the resumer (cube-lifecycle-manager/internal/resumer/resumer.go), which watches the state key for wake-up signals.

When a paused sandbox receives new activity, the resumer:

1. Clears the pause flag in Redis
2. Signals Cubelet to re-attach the volume
3. Restores the sandbox to running state

if client.HasPendingWork(sandboxID) {
    client.Resume(ctx, sandboxID)  // clears paused flag, re‑attaches volume
}

The state transition maintains data consistency by ensuring the volume is fully re-attached before the sandbox process resumes execution.

Cleanup and Resource Reclamation

When a sandbox terminates—whether through normal completion, timeout, or explicit deletion—Cubelet executes a deterministic cleanup sequence.

Template Cleanup and Volume Removal

Cubelet invokes the template cleanup routines in Cubelet/storage/template_cleanup.go to remove all associated resources:

• Sandbox volumes allocated in local.go
• Filesystem snapshots
• Host-directory bindings and mount points

This cleanup ensures that storage pools remain unfragmented and that no orphaned resources persist in the guest VM after sandbox termination.

Summary

• Cubelet manages the complete sandbox lifecycle inside each guest VM, from volume creation in Cubelet/storage/pool.go to final cleanup in template_cleanup.go.
• State coordination uses Redis keys (MetaKey and StateKey) with atomic SET NX EX operations to guarantee exclusive ownership between Cubelet and the lifecycle manager.
• Auto-pause functionality is implemented by the sweeper (cube-lifecycle-manager/internal/sweeper/sweeper.go) based on TimeoutSeconds and AutoPause flags stored in the registry.
• Resume operations are handled by the resumer (cube-lifecycle-manager/internal/resumer/resumer.go), which clears pause states and re-attaches volumes.
• Storage operations remain local to the VM through Cubelet/storage/local.go, while the master-side components manage only lightweight metadata and state transitions.

Frequently Asked Questions

What happens if Cubelet crashes during a sandbox pause operation?

If Cubelet crashes during a pause, the Redis state key remains in an inconsistent state until the lifecycle manager detects the failure. Because the sweeper uses the Bootstrap method to read all sandbox metadata on startup, it can identify sandboxes with stale state keys and either resume them or trigger reclamation based on the last recorded heartbeat timestamp. The SandboxLifecycleMeta persists in the registry regardless of Cubelet's availability.

How does Cubelet ensure exclusive control of a sandbox?

Cubelet acquires exclusive control through the AcquireState method in cube-lifecycle-manager/internal/redisstream/stream.go, which executes a Redis SET command with NX (only if not exists) and EX (expiration) flags. This atomic operation prevents multiple managers from simultaneously controlling the same sandbox, ensuring deterministic state transitions during pause, resume, and delete operations.

Can the auto-pause timeout be configured per sandbox?

Yes, each sandbox stores its own TimeoutSeconds value in the SandboxLifecycleMeta struct defined in cube-lifecycle-manager/internal/registry/registry.go. The sweeper evaluates this field individually for each sandbox when determining idle timeouts, allowing different sandboxes to have different auto-pause policies within the same Cubelet instance.

What is the difference between the sweeper and resumer components?

The sweeper (cube-lifecycle-manager/internal/sweeper/sweeper.go) proactively monitors sandbox activity and initiates pause sequences when idle timeouts occur, while the resumer (cube-lifecycle-manager/internal/resumer/resumer.go) reacts to incoming work signals by clearing pause flags and re-attaching volumes. The sweeper handles the "sleep" transition, and the resumer handles the "wake" transition, creating a bidirectional lifecycle management system.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}