# Cubelet Node-Level Sandbox Scheduling and Lifecycle Management

> Discover Cubelet node-level sandbox scheduling and lifecycle management. Learn how Cubelet handles storage and network provisioning with automatic rollback for efficient container orchestration.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-03

---

**Cubelet manages the complete sandbox lifecycle on each compute node through a workflow engine that serializes operations using semaphore-based concurrency limits, executes pluggable flow steps for storage and network provisioning, and automatically triggers rollback flows when creation fails.**

The TencentCloud/CubeSandbox project implements a sophisticated node-level orchestration system through its **Cubelet** component. Unlike traditional container schedulers that operate at the cluster level, **Cubelet node-level sandbox scheduling and lifecycle management** provides fine-grained control over sandbox creation, runtime execution, and teardown directly on the compute node. This architecture ensures resource constraints are respected while maintaining reliability through automatic failover mechanisms.

## Workflow Engine Architecture

The core of Cubelet's scheduling capability is the **Workflow Engine**, a plugin-based state machine defined in [`Cubelet/plugins/workflow/engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/engine.go). This engine registers discrete flows—init, create, destroy, and cleanup—and orchestrates their execution through a unified control plane.

The architecture consists of four distinct layers:

- **Workflow Engine**: Maintains a registry of flows and executes them with semaphore-based concurrency limiting. Main source: [`Cubelet/plugins/workflow/engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/engine.go)
- **Flow Registration**: Reads TOML/JSON workflow descriptions in [`Cubelet/plugins/workflow/plugin/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/plugin/plugin.go) to resolve plugin names into concrete `Flow` implementations
- **Concrete Implementations**: Execute node-level actions such as storage provisioning and network preparation, found in files like [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go)
- **Metrics and Failover**: Collects per-flow metrics and triggers automatic rollback via [`Cubelet/plugins/workflow/metric.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/metric.go)

## Concurrency Control and Scheduling

Cubelet implements **per-flow concurrency limiting** to prevent node resource exhaustion. The engine maintains a named semaphore (`semaphore.Limiter`) for each registered flow, acquired before execution begins.

In [`Cubelet/plugins/workflow/engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/engine.go) (lines 37-41), the engine checks the limiter before processing any request:

```go
if !flow.Limiter.TryAcquire() {
    return ret.Errorf(errorcode.ErrorCode_ConcurrentFailed,
                     "flow [%s] exceed limited", flow.ID())
}
defer flow.Limiter.Release()

```

Administrators tune per-node parallelism through the `concurrent` field in the workflow configuration. The limiter initializes via `SetFlowLimit` and `semaphore.NewLimiter` during plugin initialization, ensuring that storage, network, and runtime operations respect node capacity boundaries.

## Sandbox Lifecycle Execution

### Create Flow

The creation process begins at `Engine.Create(ctx, opts)`, which forwards to `run(flow_create, …)` as implemented in [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go) (lines 44-46). The `parallelRunSteps` function orchestrates execution by spawning goroutines for each action within a step, invoking the specific `Create` method of concrete `Flow` implementations such as `storage.Local` or `network.Local`.

The execution path proceeds through distinct phases:

1. Acquire the create-flow semaphore
2. Execute provisioned steps (storage → network → runtime)
3. Record metrics via [`metric.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/metric.go)
4. Release semaphore upon completion

### Destroy and Cleanup Flows

The destroy flow mirrors the creation path but invokes `Flow.Destroy` methods. Following successful destruction, the engine may trigger a dedicated **cleanup flow** registered under the `GCID` identifier.

The `Engine.cleanUp` method (lines 94-100 of [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go)) handles post-destruction resource reclamation, invoked either automatically after destroy operations or through explicit `Engine.CleanUp` calls for periodic garbage collection.

## Automatic Failover and Rollback

When sandbox creation encounters errors, Cubelet provides **automatic rollback** capabilities. If the request's `Failover` flag is set to `true` and any step returns an error, the engine immediately schedules a rollback flow.

In [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go) (lines 58-63), error handling after `parallelRunSteps` checks the failover condition and invokes the destroy flow for the same sandbox ID:

```go
// Error handling triggers failover
if err != nil && ctx.Failover {
    engine.failover(ctx.SandboxID)
}

```

This guarantees that partially created resources—such as allocated storage volumes or network namespaces—are torn down before the error returns to the caller, maintaining node hygiene and preventing resource leaks.

## Plugin Registration and Flow Composition

The workflow definition is supplied via TOML/JSON configuration (field `Flows`). During plugin initialization in [`Cubelet/plugins/workflow/plugin/plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/plugin/plugin.go), the system builds workflow objects from these definitions.

The `InitFn` (lines 55-63 and 98-104) constructs the engine and registers flows:

```go
engine := &workflow.Engine{}
for k, s := range config.Flows {
    flow := &workflow.Workflow{Name: k, MaxConcurrent: s.MaxConcurrent}
    // Translate step action names to concrete Flow objects
    flow.Limiter = semaphore.NewLimiter(s.MaxConcurrent)
    engine.AddFlow(k, flow)
}

```

Using the containerd plugin registry (`github.com/containerd/plugin/registry`), each `Flow` implementation registers under a unique name. If a configuration references an undefined action, the engine aborts with a clear error (lines 68-76 of [`plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/plugin.go)), preventing misconfiguration at startup rather than runtime.

## Practical Example: Creating a Sandbox

The following client-side code demonstrates invoking the create flow through Cubelet's API:

```go
// Client initiates sandbox creation
createCtx := &workflow.CreateContext{
    BaseWorkflowInfo: workflow.BaseWorkflowInfo{SandboxID: "sb-123"},
    ReqInfo:          runReq,          // User request parameters
    Failover:         true,            // Enable automatic rollback
}
err := engine.Create(ctx, createCtx)

```

Internally, the engine executes:
1. Semaphore acquisition for the create flow
2. Parallel execution of configured steps (storage provisioning, network setup, runtime initialization)
3. Automatic destroy flow invocation if any step fails and `Failover` is enabled

## Summary

- **Cubelet node-level sandbox scheduling and lifecycle management** operates as a flow-based executor rather than a traditional pod scheduler, managing the complete lifecycle from init through cleanup.
- **Concurrency control** is enforced via per-flow semaphores in [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go), with limits configurable through the `concurrent` field to prevent node overload.
- **Automatic failover** triggers destroy flows when creation fails (if `Failover: true`), ensuring resource cleanup via the rollback mechanism.
- **Plugin architecture** allows new lifecycle steps to be added by implementing the `Flow` interface and registering in configuration, without modifying core engine logic in [`Cubelet/plugins/workflow/engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/workflow/engine.go).
- **Cleanup flows** registered under `GCID` run after destruction or during periodic GC to reclaim node resources.

## Frequently Asked Questions

### What is the difference between Cubelet and CubeMaster scheduling?

Cubelet handles node-level sandbox lifecycle management and execution, while CubeMaster operates at the cluster level for placement decisions. Cubelet owns the actual creation, teardown, and resource cleanup on individual compute nodes through its workflow engine, whereas CubeMaster decides which node receives a sandbox based on resource availability and scheduling policies.

### How does Cubelet handle concurrent sandbox creation requests?

Cubelet uses a semaphore-based limiter (`semaphore.Limiter`) defined in [`Cubelet/pkg/semaphore/limiter.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/pkg/semaphore/limiter.go) and enforced in [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go). Each flow type (create, destroy, cleanup) maintains its own named semaphore with configurable limits. When requests exceed the `MaxConcurrent` threshold, the engine returns `ErrorCode_ConcurrentFailed` immediately, preventing node resource exhaustion.

### What happens when a sandbox creation fails in Cubelet?

If the `Failover` flag is set to `true` in the `CreateContext`, Cubelet automatically triggers a rollback. The engine invokes the destroy flow for the same `SandboxID` (lines 58-63 of [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go)), tearing down any partially created resources such as storage volumes or network interfaces before returning the error to the caller.

### How are new lifecycle steps added to Cubelet?

New steps are added by implementing the `Flow` interface in a concrete type (e.g., [`Cubelet/storage/local.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/local.go)), registering the implementation with the containerd plugin registry under a unique name, and referencing that name in the workflow configuration TOML/JSON. The `InitFn` in [`plugin.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/plugin.go) resolves these names to objects during startup, allowing extension without modifying [`engine.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/engine.go).