Cubelet Node-Level Sandbox Scheduling and Lifecycle Management

Cubelet manages the complete sandbox lifecycle on each compute node through a workflow engine that serializes operations using semaphore-based concurrency limits, executes pluggable flow steps for storage and network provisioning, and automatically triggers rollback flows when creation fails.

The TencentCloud/CubeSandbox project implements a sophisticated node-level orchestration system through its Cubelet component. Unlike traditional container schedulers that operate at the cluster level, Cubelet node-level sandbox scheduling and lifecycle management provides fine-grained control over sandbox creation, runtime execution, and teardown directly on the compute node. This architecture ensures resource constraints are respected while maintaining reliability through automatic failover mechanisms.

Workflow Engine Architecture

The core of Cubelet's scheduling capability is the Workflow Engine, a plugin-based state machine defined in Cubelet/plugins/workflow/engine.go. This engine registers discrete flows—init, create, destroy, and cleanup—and orchestrates their execution through a unified control plane.

The architecture consists of four distinct layers:

• Workflow Engine: Maintains a registry of flows and executes them with semaphore-based concurrency limiting. Main source: Cubelet/plugins/workflow/engine.go
• Flow Registration: Reads TOML/JSON workflow descriptions in Cubelet/plugins/workflow/plugin/plugin.go to resolve plugin names into concrete Flow implementations
• Concrete Implementations: Execute node-level actions such as storage provisioning and network preparation, found in files like Cubelet/storage/local.go
• Metrics and Failover: Collects per-flow metrics and triggers automatic rollback via Cubelet/plugins/workflow/metric.go

Concurrency Control and Scheduling

Cubelet implements per-flow concurrency limiting to prevent node resource exhaustion. The engine maintains a named semaphore (semaphore.Limiter) for each registered flow, acquired before execution begins.

In Cubelet/plugins/workflow/engine.go (lines 37-41), the engine checks the limiter before processing any request:

if !flow.Limiter.TryAcquire() {
    return ret.Errorf(errorcode.ErrorCode_ConcurrentFailed,
                     "flow [%s] exceed limited", flow.ID())
}
defer flow.Limiter.Release()

Administrators tune per-node parallelism through the concurrent field in the workflow configuration. The limiter initializes via SetFlowLimit and semaphore.NewLimiter during plugin initialization, ensuring that storage, network, and runtime operations respect node capacity boundaries.

Sandbox Lifecycle Execution

Create Flow

The creation process begins at Engine.Create(ctx, opts), which forwards to run(flow_create, …) as implemented in engine.go (lines 44-46). The parallelRunSteps function orchestrates execution by spawning goroutines for each action within a step, invoking the specific Create method of concrete Flow implementations such as storage.Local or network.Local.

The execution path proceeds through distinct phases:

1. Acquire the create-flow semaphore
2. Execute provisioned steps (storage → network → runtime)
3. Record metrics via metric.go
4. Release semaphore upon completion

Destroy and Cleanup Flows

The destroy flow mirrors the creation path but invokes Flow.Destroy methods. Following successful destruction, the engine may trigger a dedicated cleanup flow registered under the GCID identifier.

The Engine.cleanUp method (lines 94-100 of engine.go) handles post-destruction resource reclamation, invoked either automatically after destroy operations or through explicit Engine.CleanUp calls for periodic garbage collection.

Automatic Failover and Rollback

When sandbox creation encounters errors, Cubelet provides automatic rollback capabilities. If the request's Failover flag is set to true and any step returns an error, the engine immediately schedules a rollback flow.

In engine.go (lines 58-63), error handling after parallelRunSteps checks the failover condition and invokes the destroy flow for the same sandbox ID:

// Error handling triggers failover
if err != nil && ctx.Failover {
    engine.failover(ctx.SandboxID)
}

This guarantees that partially created resources—such as allocated storage volumes or network namespaces—are torn down before the error returns to the caller, maintaining node hygiene and preventing resource leaks.

Plugin Registration and Flow Composition

The workflow definition is supplied via TOML/JSON configuration (field Flows). During plugin initialization in Cubelet/plugins/workflow/plugin/plugin.go, the system builds workflow objects from these definitions.

The InitFn (lines 55-63 and 98-104) constructs the engine and registers flows:

engine := &workflow.Engine{}
for k, s := range config.Flows {
    flow := &workflow.Workflow{Name: k, MaxConcurrent: s.MaxConcurrent}
    // Translate step action names to concrete Flow objects
    flow.Limiter = semaphore.NewLimiter(s.MaxConcurrent)
    engine.AddFlow(k, flow)
}

Using the containerd plugin registry (github.com/containerd/plugin/registry), each Flow implementation registers under a unique name. If a configuration references an undefined action, the engine aborts with a clear error (lines 68-76 of plugin.go), preventing misconfiguration at startup rather than runtime.

Practical Example: Creating a Sandbox

The following client-side code demonstrates invoking the create flow through Cubelet's API:

// Client initiates sandbox creation
createCtx := &workflow.CreateContext{
    BaseWorkflowInfo: workflow.BaseWorkflowInfo{SandboxID: "sb-123"},
    ReqInfo:          runReq,          // User request parameters
    Failover:         true,            // Enable automatic rollback
}
err := engine.Create(ctx, createCtx)

Internally, the engine executes:

1. Semaphore acquisition for the create flow
2. Parallel execution of configured steps (storage provisioning, network setup, runtime initialization)
3. Automatic destroy flow invocation if any step fails and Failover is enabled

Summary

• Cubelet node-level sandbox scheduling and lifecycle management operates as a flow-based executor rather than a traditional pod scheduler, managing the complete lifecycle from init through cleanup.
• Concurrency control is enforced via per-flow semaphores in engine.go, with limits configurable through the concurrent field to prevent node overload.
• Automatic failover triggers destroy flows when creation fails (if Failover: true), ensuring resource cleanup via the rollback mechanism.
• Plugin architecture allows new lifecycle steps to be added by implementing the Flow interface and registering in configuration, without modifying core engine logic in Cubelet/plugins/workflow/engine.go.
• Cleanup flows registered under GCID run after destruction or during periodic GC to reclaim node resources.

Frequently Asked Questions

What is the difference between Cubelet and CubeMaster scheduling?

Cubelet handles node-level sandbox lifecycle management and execution, while CubeMaster operates at the cluster level for placement decisions. Cubelet owns the actual creation, teardown, and resource cleanup on individual compute nodes through its workflow engine, whereas CubeMaster decides which node receives a sandbox based on resource availability and scheduling policies.

How does Cubelet handle concurrent sandbox creation requests?

Cubelet uses a semaphore-based limiter (semaphore.Limiter) defined in Cubelet/pkg/semaphore/limiter.go and enforced in engine.go. Each flow type (create, destroy, cleanup) maintains its own named semaphore with configurable limits. When requests exceed the MaxConcurrent threshold, the engine returns ErrorCode_ConcurrentFailed immediately, preventing node resource exhaustion.

What happens when a sandbox creation fails in Cubelet?

If the Failover flag is set to true in the CreateContext, Cubelet automatically triggers a rollback. The engine invokes the destroy flow for the same SandboxID (lines 58-63 of engine.go), tearing down any partially created resources such as storage volumes or network interfaces before returning the error to the caller.

How are new lifecycle steps added to Cubelet?

New steps are added by implementing the Flow interface in a concrete type (e.g., Cubelet/storage/local.go), registering the implementation with the containerd plugin registry under a unique name, and referencing that name in the workflow configuration TOML/JSON. The InitFn in plugin.go resolves these names to objects during startup, allowing extension without modifying engine.go.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}