How Containerd Interacts with CubeSandbox: The Shim v2 Architecture Explained

Containerd interacts with CubeSandbox exclusively through CubeShim, a custom containerd Shim v2 implementation written in Rust that bridges OCI container management with KVM MicroVM orchestration while containerd handles image pulling and snapshot storage.

CubeSandbox is an open-source serverless container runtime developed by Tencent Cloud that isolates workloads using MicroVMs. Unlike traditional container runtimes that use runc, CubeSandbox leverages containerd as its image and storage manager but delegates actual sandbox execution to a specialized shim layer. Understanding this interaction requires examining the specific integration points in the Cubelet node agent and the CubeShim binary.

The CubeShim Bridge: Containerd's Gateway to MicroVMs

CubeSandbox components operate as a tightly coupled stack on each node, with containerd touching the sandbox only through CubeShim. This Rust-based binary implements the containerd Shim v2 API, acting as a translation layer between containerd's task management and the CubeHypervisor's MicroVM lifecycle operations.

When containerd receives a request to create a container, it invokes the registered shim binary (containerd-shim-cube-rs) rather than runc. The shim receives standard OCI runtime calls but translates them into VM creation commands. In CubeShim/shim/src/main.rs, the shim implements the Create and Start methods to prepare root filesystems, memory files, and kernel images before booting the MicroVM via CubeHypervisor.

Runtime Registration and Configuration

Before containerd can delegate tasks to CubeSandbox, the Cubelet process must register CubeShim as a valid runtime. This registration occurs during node initialization in Cubelet/services/server/plugins_compat.go.

The registration code calls registry.Register(&plugin.Plugin{...}) with the specific runtime name io.containerd.cube.rs. This identifier allows containerd to route tasks to the correct shim binary when users specify the Cube runtime.

To enable this integration, configure containerd with the runtime definition:

# /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.rs"
  binary = "/usr/local/bin/containerd-shim-cube-rs"

The binary path points to the pre-built Rust shim executable, typically installed by the one-click deployment scripts at deploy/one-click/config-cube.toml.

Image Management and OCI Integration

While CubeShim handles VM execution, containerd retains responsibility for OCI image management. Cubelet utilizes the standard containerd client library to pull images, manage layer caching, and prepare root filesystems in the snapshot store.

In Cubelet/services/images/service.go, the image service constructs a containerd client using github.com/containerd/containerd/v2 to fetch images and expose them to the sandbox creation path. This delegation allows CubeSandbox to leverage containerd's registry authentication, layer deduplication, and snapshotter plugins without reimplementing image handling logic.

The workflow follows this sequence:

• Cubelet requests an image pull via the containerd client
• Containerd caches layers in its snapshot store
• CubeShim accesses these prepared rootfs bundles when creating the MicroVM

Lifecycle Management: From Create to Pause/Resume

Once a sandbox is running, CubeShim maintains a bidirectional vsock channel to the Cube agent—a Go process running inside the guest VM. This channel uses the ttrpc protocol to handle lifecycle commands from the control plane.

The auto-pause functionality demonstrates the full interaction chain:

1. The side-car attached to CubeProxy publishes a pause event to Redis when the sandbox becomes idle
2. Cubelet receives the event and forwards a pause request to containerd
3. Containerd invokes task.Pause() on the CubeShim
4. CubeShim takes an in-place snapshot of the VM state and freezes the MicroVM
5. On the next request, a resume RPC flows through CubeMaster → Cubelet → containerd → CubeShim
6. CubeShim restores the VM from the snapshot in Cubelet/services/cubebox/update.go

This architecture allows containerd to treat MicroVMs as standard container tasks while CubeShim handles the virtualization complexity.

Code Examples

Configuring Containerd for CubeShim

Enable the Cube runtime by adding this configuration to /etc/containerd/config.toml:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.rs"
  binary = "/usr/local/bin/containerd-shim-cube-rs"

This configuration tells containerd to invoke the CubeShim binary for any pod specifying the Cube runtime class.

Creating a Sandbox via the Containerd Client

The following Go code demonstrates how Cubelet uses the containerd client to create a sandbox with the Cube runtime:

import (
    "context"
    "github.com/containerd/containerd/v2"
    "github.com/containerd/containerd/v2/pkg/namespaces"
)

func createSandbox(ctx context.Context, imgRef string) (containerd.Task, error) {
    client, err := containerd.New("/run/containerd/containerd.sock")
    if err != nil {
        return nil, err
    }
    defer client.Close()

    ctx = namespaces.WithNamespace(ctx, "default")

    image, err := client.Pull(ctx, imgRef, containerd.WithPullUnpack)
    if err != nil {
        return nil, err
    }

    container, err := client.NewContainer(
        ctx,
        "my-sandbox",
        containerd.WithImage(image),
        containerd.WithRuntime("io.containerd.cube.rs", nil),
    )
    if err != nil {
        return nil, err
    }

    task, err := container.NewTask(ctx, containerd.EmptyIO)
    if err != nil {
        return nil, err
    }
    
    if err := task.Start(ctx); err != nil {
        return nil, err
    }
    return task, nil
}

The critical line is containerd.WithRuntime("io.containerd.cube.rs", nil), which triggers containerd to invoke CubeShim rather than the default runc runtime.

Handling Pause and Resume Operations

The auto-scaling mechanism interacts with containerd's task API to freeze and restore sandboxes:

// Pause the sandbox (triggered by idle timeout)
if err := task.Pause(ctx); err != nil {
    return err
}

// Resume the sandbox (triggered by incoming request)
if err := task.Start(ctx); err != nil {
    return err
}

These methods are defined in the standard containerd Task interface but are implemented by CubeShim to perform VM snapshotting and restoration via the CubeHypervisor.

Key Source Files and Implementation Details

Understanding the containerd integration requires examining these specific files in the TencentCloud/CubeSandbox repository:

• Cubelet/services/server/plugins_compat.go – Registers the io.containerd.cube.rs runtime with containerd's plugin registry
• CubeShim/shim/src/main.rs – Implements the containerd Shim v2 API including Create, Start, and Delete task operations
• Cubelet/services/images/service.go – Demonstrates containerd client usage for OCI image pulling and management
• Cubelet/services/cubebox/update.go – Contains the pause/resume logic that forwards lifecycle commands from containerd to the shim
• deploy/one-click/config-cube.toml – Production configuration template for enabling the Cube runtime in containerd
• docs/architecture/overview.md – Visual and textual overview of the data-plane interaction between containerd, CubeShim, and the hypervisor

Summary

• CubeShim acts as the sole interface between containerd and CubeSandbox, implementing the Shim v2 API in Rust to translate container tasks into MicroVM operations.
• Containerd manages OCI images and snapshots while delegating execution to the io.containerd.cube.rs runtime registered by Cubelet.
• The registration occurs in plugins_compat.go, where Cubelet announces the shim binary location to containerd's plugin system.
• Lifecycle operations including auto-pause and auto-resume flow through containerd's Task API to CubeShim, which coordinates with CubeHypervisor for VM snapshot management.
• Bidirectional vsock channels enable communication between CubeShim and the Cube agent inside the guest, using ttrpc for control-plane commands.

Frequently Asked Questions

What is CubeShim and why does CubeSandbox need it?

CubeShim is a custom containerd Shim v2 implementation written in Rust that serves as the bridge between containerd and CubeSandbox's MicroVM architecture. CubeSandbox needs this shim because containerd expects to communicate with runc-compatible runtimes by default, but CubeSandbox uses KVM-based MicroVMs for isolation. The shim translates standard OCI runtime calls (Create, Start, Kill) into VM lifecycle operations while presenting a compatible interface to containerd.

Does CubeSandbox use containerd for image pulling?

Yes, CubeSandbox delegates all OCI image operations to containerd. The Cubelet component uses the standard containerd Go client (github.com/containerd/containerd/v2) to pull images from registries, handle layer caching, and store unpacked root filesystems in containerd's snapshot store. This design allows CubeSandbox to inherit containerd's robust image management capabilities including registry authentication and layer deduplication without reimplementing these features.

How does pause/resume work with containerd in CubeSandbox?

When a sandbox becomes idle, a side-car process publishes a pause event to Redis. Cubelet receives this event and calls task.Pause() on the containerd task, which forwards the request to CubeShim. The shim then instructs CubeHypervisor to take an in-place snapshot of the VM and freeze it. For resumption, a resume RPC flows through the control plane to Cubelet, which invokes task.Start(), triggering CubeShim to restore the VM from its snapshot and reestablish the vsock connection to the Cube agent.

Is CubeSandbox compatible with standard Kubernetes CRI implementations?

CubeSandbox is compatible with Kubernetes through containerd's CRI plugin, but it requires the CubeShim runtime to be registered as io.containerd.cube.rs. Standard runc containers use the io.containerd.runc.v2 runtime, while CubeSandbox workloads must specify the Cube runtime class in their pod specifications. The CRI implementation in containerd handles the routing to the appropriate shim based on the runtime class name specified in the Kubernetes pod spec.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}