Lifecycle Management of the Cubelet Local Scheduling Component in CubeSandbox

Cubelet manages the complete lifecycle of sandbox instances on a single compute node, handling creation, execution, pause, resume, snapshot, and destruction through tight integration with containerd, CubeCoW, and the network-agent.

Cubelet serves as the node-local scheduling agent in TencentCloud's CubeSandbox architecture, owning the entire lifecycle of every sandbox instance that executes on a compute node. This component orchestrates the transition between states—from initial creation through final destruction—while coordinating with underlying virtualization and storage systems. Understanding the lifecycle management of the Cubelet local scheduling component is essential for operators deploying micro-VM workloads at scale.

The Six Lifecycle Stages of Cubelet

According to docs/architecture/overview.md, Cubelet implements a deterministic state machine that transitions sandbox instances through six distinct phases. Each stage involves specific resource allocation and deallocation operations that ensure consistent runtime behavior.

Create Stage

During the Create stage, Cubelet receives a provisioning request from CubeMaster or CubeAPI. The component initiates three parallel workstreams:

1. Image acquisition – Pulls the container image through containerd and validates layer integrity
2. Storage preparation – Allocates root filesystem resources using CubeCoW (Copy-on-Write) volumes for efficient disk utilization
3. Network provisioning – Acquires a TAP device file descriptor via the network-agent service

This stage completes when the micro-VM environment is fully prepared but not yet executing.

Run Stage

The Run stage transitions the sandbox from a prepared state to active execution. Cubelet launches the KVM micro-VM, initializes the sandbox's PID 1 (cube-agent), and starts the user-defined workload process. Health checks begin immediately, with Cubelet reporting status to CubeMaster via periodic RPCs to enable cluster-wide visibility.

Pause Stage

When instructed by the control plane, Cubelet enters the Pause stage by invoking the KVM pause API. This freezes the VM state, halts CPU execution, and prevents new I/O processing while maintaining memory contents. Cubelet updates its internal state tables to reflect the paused condition and stops forwarding health check heartbeats until the resume operation completes.

Resume Stage

The Resume stage thaws a previously paused VM. Cubelet executes the KVM resume operation, re-attaches storage layers, and re-binds the TAP device through the network-agent. The workload continues execution from its exact pre-pause state, ensuring no data loss or connection disruption during the pause/resume cycle.

Snapshot Stage

During the Snapshot stage, Cubelet captures a point-in-time image of the sandbox's disk and memory state. This operation leverages CubeCoW snapshot APIs to create persistent storage layers that can be used for cloning new sandboxes or rollback operations. Cubelet writes snapshot metadata to CubeMaster, maintaining the lineage relationship between the original instance and its snapshot derivatives.

Destroy Stage

The Destroy stage represents the terminal lifecycle phase. Cubelet performs comprehensive resource cleanup:

• Releases TAP devices via network-agent
• Removes CubeCoW storage layers and unmounts filesystems
• Destroys cgroups and kernel namespaces
• Deregisters the sandbox from the internal registry
• Reports final status to CubeMaster

Integration Architecture

The lifecycle management capabilities depend on three critical subsystem integrations documented in the configuration at Cubelet/config/config.toml.

containerd Integration

Cubelet delegates all container image operations to containerd, the industry-standard container runtime. During the Create stage, Cubelet invokes containerd's image pull APIs to fetch and unpack layer tarballs into the local content store. This integration ensures compatibility with standard OCI image formats while maintaining the security boundaries required for micro-VM isolation.

CubeCoW Storage Layer

CubeCoW provides the copy-on-write filesystem that underpins all storage lifecycle operations. Located logically in Cubelet/internal/storage.go, this subsystem enables:

• Fast snapshot creation through block-level copy-on-write mechanisms
• Efficient cloning of sandboxes from existing snapshots
• Layered storage that minimizes disk duplication across similar workloads

The CubeCoW integration allows the Snapshot stage to complete in milliseconds rather than minutes, as only changed blocks require copying.

network-agent Coordination

Network resource lifecycle management occurs through the network-agent service, with integration code residing in Cubelet/internal/network.go. This component:

• Allocates TAP file descriptors during Create and Resume stages
• Enforces network policy rules during Run stage
• Releases virtual network interfaces during Destroy stage

The separation of network concerns into a dedicated agent allows Cubelet to remain agnostic of specific SDN implementations while ensuring consistent TAP device management.

Implementation Deep Dive

While the specific implementation details reside in the internal source tree, the architecture documentation identifies several key files responsible for lifecycle orchestration:

• Cubelet/internal/scheduler.go – Contains the core state machine implementing the create-run-pause-resume-snapshot-destroy pipeline
• Cubelet/internal/network.go – Handles TAP FD acquisition and release coordination with the network-agent
• Cubelet/internal/storage.go – Implements CubeCoW volume lifecycle hooks and snapshot management
• Cubelet/config/config.toml – Centralizes runtime parameters including image pull timeouts, snapshot retention policies, and network configuration

Practical Example: Orchestrating Lifecycle Transitions

The following Go SDK example demonstrates how client applications interact with Cubelet's lifecycle management through CubeAPI. Each method call triggers the corresponding stage transition in the node-local scheduler:

package main

import (
    "context"
    "log"

    "github.com/TencentCloud/CubeSandbox/CubeAPI/client"
    "github.com/TencentCloud/CubeSandbox/CubeAPI/types"
)

func main() {
    // Initialize client pointing to CubeMaster
    c, err := client.NewClient("http://cube-master:8080")
    if err != nil { 
        log.Fatalf("client init: %v", err) 
    }

    // Create stage: Cubelet allocates storage, pulls image, acquires TAP
    sandbox, err := c.SandboxCreate(context.Background(), &types.SandboxCreateRequest{
        Image: "docker.io/library/ubuntu:latest",
        Cmd:   []string{"/bin/bash"},
    })
    if err != nil { 
        log.Fatalf("create failed: %v", err) 
    }
    log.Printf("Sandbox %s created", sandbox.ID)

    // Pause stage: KVM freeze, I/O halt
    if err := c.SandboxPause(context.Background(), sandbox.ID); err != nil {
        log.Fatalf("pause failed: %v", err)
    }
    log.Println("Sandbox paused")

    // Resume stage: KVM thaw, resource re-attachment
    if err := c.SandboxResume(context.Background(), sandbox.ID); err != nil {
        log.Fatalf("resume failed: %v", err)
    }
    log.Println("Sandbox resumed")

    // Snapshot stage: CubeCoW point-in-time capture
    snap, err := c.SandboxSnapshot(context.Background(), sandbox.ID, &types.SnapshotRequest{
        Name: "checkpoint-v1",
    })
    if err != nil {
        log.Fatalf("snapshot failed: %v", err)
    }
    log.Printf("Snapshot created: %s", snap.ID)

    // Destroy stage: Complete resource cleanup
    if err := c.SandboxDestroy(context.Background(), sandbox.ID); err != nil {
        log.Fatalf("destroy failed: %v", err)
    }
    log.Println("Sandbox destroyed")
}

Each API operation maps directly to Cubelet's internal lifecycle stages, with the SDK abstracting the RPC communication between CubeMaster and the node-local scheduler.

Summary

• Cubelet functions as a self-contained scheduling agent on each compute node, eliminating centralized bottlenecks for lifecycle operations
• The component implements a six-stage lifecycle (Create → Run → Pause → Resume → Snapshot → Destroy) that covers the complete sandbox existence
• Integration with containerd, CubeCoW, and network-agent provides specialized handling for images, storage, and networking without bloating the core scheduler
• State persistence during Pause and Snapshot stages enables advanced use cases like live migration and instant cloning
• Resource cleanup during the Destroy stage ensures no orphaned TAP devices, storage layers, or cgroups remain on the node

Frequently Asked Questions

How does Cubelet handle resource cleanup if a node crashes mid-lifecycle?

Cubelet maintains persistent state records that survive process restarts. Upon restart, Cubelet reconciles its internal registry against actual system resources (TAP devices, KVM processes, CubeCoW volumes) and performs garbage collection on orphaned resources. The component also reports sync status to CubeMaster, allowing the control plane to identify sandboxes that may require manual remediation.

Can Cubelet pause and resume sandboxes across different physical nodes?

No, the Pause and Resume stages are node-local operations. While Cubelet can freeze a VM's state, transferring that state to another node requires the Snapshot stage followed by Create on the target node. True live migration would involve copying memory pages and disk state between nodes, which is not part of the current lifecycle implementation described in docs/architecture/overview.md.

What is the performance impact of the Snapshot stage on running workloads?

The Snapshot stage utilizes CubeCoW copy-on-write mechanisms to minimize performance impact. Since snapshots are implemented as block-level references rather than full copies, the operation typically completes in milliseconds with minimal I/O overhead. However, the first write to any block after snapshotting incurs a minor latency penalty as the original block is copied to the snapshot layer.

How does Cubelet ensure network isolation during the Resume stage?

During Resume, Cubelet re-acquires the TAP file descriptor from network-agent and re-applies all network policies before unpausing the VM. This ensures that network traffic cannot enter or exit the sandbox until the security rules are fully re-established. The network-agent integration in Cubelet/internal/network.go handles the atomic re-binding of interfaces to prevent race conditions.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}