CubeSandbox Containerd Shim v2 Integration Approach: Architecture and Implementation

CubeSandbox implements containerd shim v2 through a Rust-based binary (containerd-shim-cube-rs) that registers as a containerd plugin, communicates with guest VMs via vsock, and manages sandbox lifecycle through standard ttrpc services while maintaining the binary at /usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs.

TencentCloud/CubeSandbox leverages the containerd shim v2 API to run each sandbox's init process as a dedicated shim instance. This integration approach allows the platform to manage VM-based workloads using standard containerd interfaces while adding Cube-specific communication channels for guest agent interaction.

Shim Binary Implementation

The core of the integration is the containerd-shim-cube-rs binary implemented in Rust. Located in CubeShim/shim/src/main.rs, this program implements the complete Shim v2 API including ttrpc services, state handling, and I/O forwarding. The shim acts as the parent process for each sandbox, maintaining the container lifecycle according to containerd v2 specifications while extending functionality for VM-based isolation.

The entry point parses containerd arguments (shim ID, bundle path) and establishes the server:

// Inside CubeShim/shim/src/main.rs – entry point of the shim v2.
fn main() {
    // Parse Containerd arguments (shim ID, bundle, etc.)
    let args = shim::Args::parse();
    // Register ttrpc services (state, I/O, health)
    let server = ttrpc::Server::new(...);
    // Connect to the Cube agent via vsock for log forwarding.
    let agent = AgentClient::connect(vsock_addr);
    // Run the event loop.
    server.start();
}

Plugin Registration and Discovery

CubeSandbox registers the shim through containerd's plugin registry mechanism. The integration code in Cubelet/services/server/plugins_compat.go builds the plugin list, enabling containerd to automatically discover and launch containerd-shim-cube-rs when creating new sandboxes.

This registration allows the shim to participate in containerd's health-checking subsystem via the io.containerd.grpc.v1.healthcheck plugin. Cubelet periodically queries the shim through this interface, maintaining awareness of shim health and enabling automatic recovery of crashed or stuck instances.

Binary Location and Deployment Layout

Cubelet expects the shim binary at a specific filesystem location defined by the defaultShimPath constant in Cubelet/services/cubebox/local.go (line 272). The standard path is:

/usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs

This path is hardcoded across the deployment stack. The one-click installer (deploy/one-click/install.sh), Docker images, and Kubernetes manifests all reference this location, ensuring consistency between host-mode and Kubernetes installations.

Sandbox Lifecycle and Agent Communication

When Cubelet creates a sandbox, it invokes containerd.CreateContainer with the containerd.WithShim option pointing to the default shim path. Containerd then spawns containerd-shim-cube-rs as a child process, passing the sandbox ID, bundle path, and a vsock address for communication with the Cube agent inside the VM.

The implementation in Cubelet/services/cubebox/runc_container_op.go handles this initiation:

// Cubelet creates a sandbox and tells containerd which shim to use.
// The constant defaultShimPath resolves to the binary built by “make shim”.
const defaultShimPath = "/usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs"

opts := []containerd.NewContainerOpts{
    containerd.WithSandbox(sandboxID),          // associate container with sandbox
    containerd.WithShim(defaultShimPath),       // launch our shim v2 binary
    containerd.WithImage(image),                // the rootfs image
    // ... other OCI options
}
ctr, err := client.NewContainer(ctx, sandboxID, opts...)

Once running, the shim opens a vsock channel to the agent (running as /sbin/init in the guest) to forward:

• Lifecycle commands (pause, resume, shutdown) via ttrpc
• Stdout/err streams of the init process back to the host
• Kernel console output (console=hvc0) to cube-shim-req.log

Build and Packaging

The shim binary is built using the make shim target defined in the Makefile (line 291). This target runs Cargo inside the CubeShim workspace, installs the binary into _output/bin, and copies it into the runtime layout.

# Build and install the shim (make shim target)

make shim

# Binary ends up at /usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs

The build process integrates with deployment manifests in deploy/kubernetes/images/cube-shim/Dockerfile and deploy/one-click/build-vmlinux.sh, ensuring the binary is available at the standard path for both development and production environments.

Summary

• CubeSandbox implements containerd shim v2 through the containerd-shim-cube-rs Rust binary located in CubeShim/shim/src/main.rs.
• The shim registers with containerd via the plugin registry in Cubelet/services/server/plugins_compat.go and exposes ttrpc services for lifecycle management.
• Cubelet creates sandboxes using containerd.WithShim with the path defined in Cubelet/services/cubebox/local.go (line 272), defaulting to /usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs.
• Communication between the shim and guest VM occurs over vsock, forwarding kernel console output and init process streams to cube-shim-req.log.
• The make shim target (line 291 in Makefile) compiles the binary and installs it to the runtime layout used by deployment scripts.

Frequently Asked Questions

What is the default file path for the CubeSandbox containerd shim v2 binary?

The binary is expected at /usr/local/services/cubetoolbox/cube-shim/bin/containerd-shim-cube-rs, defined by the defaultShimPath constant in Cubelet/services/cubebox/local.go (line 272). This path is used by Cubelet when invoking containerd.WithShim during sandbox creation.

How does the shim communicate with the guest VM agent?

The shim establishes a vsock channel to communicate with the Cube agent running as /sbin/init inside the guest VM. This channel forwards lifecycle commands (pause, resume, shutdown) via ttrpc, streams stdout/stderr from the init process, and captures kernel console output (console=hvc0) for logging.

What build command compiles the shim binary?

Running make shim (defined at line 291 of the Makefile) executes Cargo within the CubeShim workspace to build the Rust binary. The target installs the resulting binary into _output/bin and copies it to the standard deployment path.

How is the shim registered with containerd?

The shim registers as a containerd plugin through the code in Cubelet/services/server/plugins_compat.go, which builds the plugin list. This registration enables containerd to automatically discover and launch the shim when creating new sandboxes, and allows Cubelet to query shim health through the io.containerd.grpc.v1.healthcheck interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}