How Cubelet Manages the Lifecycle of Sandbox Instances on a Node

Cubelet registers itself as a containerd plugin and orchestrates sandbox creation, destruction, and monitoring through a workflow engine, event-driven state reconciliation, and background garbage collection.

Cubelet is the node-side agent in TencentCloud/CubeSandbox that drives the complete lifecycle of Cubebox (sandbox) instances. According to the source code, it implements a gRPC service layer that interfaces with containerd to handle everything from initial plugin registration to automated cleanup of terminated containers.

Registration and Initialization

When the Cubelet binary starts, it registers itself as a containerd plugin using the constant constants.CubeboxServicePlugin. During the init() phase, it pulls together the internal cubebox manager (local), the workflow engine, and the containerd event plugin to build a unified service object that holds all runtime handles.

In Cubelet/services/cubebox/service.go (lines 71-82), the initialization sequence establishes the connection to the underlying containerd runtime and prepares the service to accept gRPC requests for sandbox management.

Creating Sandboxes

The Create() method in service.go (lines 22-34, 36-55, 62-73) handles the initiation of new sandbox instances through a structured validation and orchestration pipeline.

Validation and Context Building

When a create request arrives, Cubelet first validates the input and fills default values using SetRunCubeSandboxRequestDefaultValue. It then constructs a workflow.CreateContext and injects distributed tracing information to ensure observability throughout the lifecycle.

Workflow Engine Orchestration

After preprocessing, the request is handed to the workflow engine via s.engine.Create. This engine orchestrates the detailed provisioning steps: pulling the container image, creating the OCI runtime sandbox, attaching resources, and starting the containers. Once the engine returns, Cubelet enriches the response with the sandbox ID, IP address, port mappings, and internal metrics covering the shim, cubebox, service, volume, and probe components.

Destroying Sandboxes

Sandbox teardown is handled by the Destroy() method in service.go (lines 62-99, 106-124), which implements a graceful shutdown protocol with optional forced cleanup.

Metadata Retrieval and Marking

The process begins by validating the sandbox ID and reading stored metadata via cubeboxMgr.cubeboxManger.Get. If the sandbox exists, Cubelet records a user-mark-deleted timestamp and applies optional clean-up options (clean_opts) to determine whether to force immediate termination.

Engine-Driven Teardown

The destruction request is then passed to s.engine.Destroy, which coordinates the actual resource release. The function logs a rich request trace using CubeLog.RequestTrace and returns a DestroyCubeSandboxResponse containing the final status and metric payloads.

Listing and Querying Sandboxes

The List() method provides visibility into running sandboxes by pulling the full list from the cubebox store via cubeboxMgr.cubeboxManger.List(). As implemented in service.go (lines 51-78), the method truncates results to maxListCubebox if necessary and applies optional filters using applyCubeSandboxFilter before returning a gRPC-compatible CubeSandbox slice.

Event-Driven State Reconciliation

Cubelet maintains accurate sandbox state through an asynchronous event monitoring system defined in Cubelet/services/cubebox/events.go (lines 38-86).

Containerd Event Subscription

An internal event monitor (eventMonitor) subscribes to containerd topics including /tasks/exit, /tasks/oom, and /tasks/paused. For each event received, the system extracts the sandbox or container ID and checks a back-off cache to prevent duplicate processing.

Event Handling and Store Updates

Valid events are forwarded to dedicated handlers (handleEvent) that update the cubebox store, propagate OOM or exit statuses, and trigger clean-up actions when necessary. This ensures the internal state remains synchronized with the actual containerd runtime state without requiring constant polling.

Background Garbage Collection

A background goroutine (destroyDeadContainers) periodically scans the cubebox store to prevent accumulation of stale terminated containers. Implemented in service.go (lines 42-60, 70-79), this cleaner identifies containers with Terminated status whose FinishedAt timestamp exceeds the configured TTL. For each expired container, it logs a warning and increments a global deadContainerCount metric before removing the entry.

Summary

• Cubelet registers as a containerd plugin (constants.CubeboxServicePlugin) during initialization in service.go
• Creation flows through validation, context building, and the workflow engine (s.engine.Create) to orchestrate image pulls and runtime setup
• Destruction marks sandboxes for deletion and invokes s.engine.Destroy for resource cleanup while emitting detailed traces
• Event monitoring captures runtime changes (exit, OOM, pause) from containerd and updates the internal store via handlers in events.go
• Garbage collection runs periodically to remove terminated containers older than the TTL, preventing store bloat

Frequently Asked Questions

How does Cubelet track sandbox status changes?

Cubelet tracks status changes through an event-driven architecture. The eventMonitor subscribes to containerd topics like /tasks/exit and /tasks/oom in events.go, then dispatches to handlers that update the cubebox store. This eliminates the need for polling and provides near real-time state synchronization.

What happens when a sandbox creation fails?

During the Create() workflow, any failure in the workflow engine (s.engine.Create) aborts the operation before the sandbox is fully registered. The engine handles image pull failures, OCI runtime errors, and resource attachment issues, returning an error response without leaving orphaned resources in the cubebox store.

Can Cubelet force-delete a stuck sandbox?

Yes. The Destroy() method accepts clean_opts that can force immediate cleanup. When the user-mark-deleted timestamp is set and force options are enabled, the workflow engine bypasses graceful shutdown timeouts and immediately terminates the OCI runtime sandbox.

Where does Cubelet store sandbox metadata?

Cubelet stores metadata through the cubeboxMgr.cubeboxManger interface implemented in local.go. This layer provides CRUD operations for CubeBox objects, which the service layer queries during Create(), Destroy(), and List() operations, and which the event monitor updates when containerd emits state changes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}