CubeShim Containerd Shim v2 Architecture: How CubeSandbox Bridges Containers and MicroVMs

CubeShim implements the containerd shim v2 specification as a Rust-based bridge that translates container lifecycle commands into MicroVM operations, communicating with an in-guest cube-agent via ttrpc over vsock.

CubeShim is the critical integration point in the TencentCloud/CubeSandbox project that allows containerd to manage sandboxed MicroVMs using standard container semantics. By implementing the containerd shim v2 API, CubeShim enables container orchestrators to treat lightweight virtual machines as regular containers without modifying the upper stack. This article examines the architecture of the CubeShim containerd shim v2 implementation, tracing how it handles container lifecycle events from the initial RPC to VM process execution.

High-Level Data Flow and Architecture

The CubeShim architecture follows a three-tier communication model that isolates the container engine from the guest runtime:

containerd
   │   (Shim v2 API)
   ▼
containerd-shim-cube-rs  ← CubeShim (this binary)
   │   (ttrpc)
   ▼
cube-agent (in-VM)
   │
   ▼
processes inside the MicroVM

containerd issues standard Shim v2 calls such as Create, Start, Exec, and Kill. CubeShim runs as the binary containerd-shim-cube-rs and implements these calls. For each request, it communicates with the cube-agent inside the guest VM via ttrpc over a vsock connection. The agent then forwards the request to the actual process runtime inside the MicroVM.

This architecture is documented in the Mermaid diagram within docs/architecture/overview.md (lines 25-27).

Core Components of the CubeShim Implementation

Shim Process and Entry Point

The shim binary entry point resides in CubeShim/shim/src/main.rs. This component boots a Tokio async runtime, parses command-line flags, and invokes shim_run::<Service>(...) to register the service implementation with the containerd-shim-rs framework. This entry point is responsible for initializing the long-running process that containerd will communicate with throughout the container lifecycle.

Service Implementation Layer

The concrete RPC handlers live in CubeShim/shim/src/service/, specifically within srv.rs and task_srv.rs. These files implement the Service struct that handles Shim v2 API methods including create, start, exec, kill, and delete. Each handler forwards requests to the in-VM cube-agent via a ttrpc client, translating containerd's expectations into VM-specific operations.

Hypervisor Integration

VM lifecycle management is abstracted in CubeShim/shim/src/hypervisor/cube_hypervisor.rs. This module wraps CubeHypervisor (built on RustVMM and KVM) to handle MicroVM creation, snapshot, restore, pause, and resume operations. When the shim receives a create request, it asks the hypervisor to instantiate a MicroVM, then extracts the vsock endpoint to hand off to the agent.

Sandbox Resource Preparation

Before VM launch, the shim prepares necessary resources in CubeShim/shim/src/container/. Files like rootfs.rs and exec.rs pre-create the root filesystem, memory file, and kernel image that the VM will use. These resources are supplied to the hypervisor during the Create RPC handling.

Logging and Statistics

Operational telemetry is managed through CubeShim/shim/src/log/stat_defer.rs. This component writes request logs and state transitions to /data/log/CubeShim/, ensuring that shim operations remain observable without affecting the containerd daemon's performance.

Common Utilities

Shared types, constants (such as SHIM_VERSION), and helper functions reside in CubeShim/shim/src/common/. These utilities provide the foundational types used across the shim's service, hypervisor, and container modules.

Container Lifecycle Implementation

Create and Start Sequence

The interaction between containerd and CubeShim follows a precise sequence when spawning a sandboxed workload:

1. containerd spawns containerd-shim-cube-rs with the sandbox ID and bundle path.
2. The shim's main.rs creates a multi-threaded Tokio runtime and calls shim_run::<Service>(...), registering the service implementation.
3. The Create RPC arrives at Service::create in srv.rs. This handler reads the sandbox bundle (rootfs, kernel, and memory file), then calls cube_hypervisor::launch_vm to instantiate the KVM MicroVM and start the cube-agent via vsock.
4. The Start RPC triggers Service::start, which forwards an Exec request over ttrpc to the agent. The agent finally executes the user's entrypoint process inside the MicroVM.
5. The shim streams the VM's stdin, stdout, and stderr back to containerd, presenting the MicroVM's output as standard container logs.

This orchestration is managed by the async service defined in CubeShim/shim/src/service/srv.rs and the state-tracking helpers in task_srv.rs.

Containerd Integration and Registration

To enable containerd to route requests to CubeShim, the runtime must be registered in /etc/containerd/config.toml. As documented in CubeShim/README.md (lines 81-87), the configuration requires:

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.v2"

The containerd-shim-cube-rs binary must be available on the host $PATH. Once configured, containerd invokes the shim whenever a sandbox using the cube runtime is created.

Why Shim v2 Architecture Matters for CubeSandbox

The Shim v2 model provides three critical advantages for CubeSandbox:

• Process Isolation: The shim remains alive after the initial containerd fork, decoupling the MicroVM's lifetime from the containerd process that launched it. This prevents containerd restarts from affecting running workloads.
• Auto-Pause and Resume: CubeShim can receive pause and resume requests from the control plane (CubeMaster) and invoke the hypervisor's snapshot and restore paths without involving containerd directly.
• Stateless Containerd Integration: By implementing the minimal subset of the Shim v2 API required for CubeSandbox, the shim keeps containerd's view of the sandbox simple while performing heavy lifting inside the VM.

Configuration and Usage Examples

Containerd Runtime Configuration

To enable the cube runtime, add the following to your containerd configuration:

# /etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.v2"
  # Optional: explicit path if not on $PATH

  #runtime_path = "/usr/local/bin/containerd-shim-cube-rs"

Creating a Sandbox via Containerd

Once configured, create a sandboxed container using the standard containerd CLI:

# Trigger containerd to launch containerd-shim-cube-rs

ctr run --runtime=cube \
    docker.io/library/ubuntu:22.04 \
    my-sandbox-id \
    /bin/bash -c "echo Hello from MicroVM"

This command initiates the full lifecycle sequence, from shim spawning to VM process execution.

Minimal Shim Client Example

For testing or custom integrations, the shim can be invoked programmatically:

use containerd_shim::asynchronous::run as shim_run;
use containerd_shim_cube_rs::service::Service;

#[tokio::main]
async fn main() {
    // Register the service under the cube runtime name
    shim_run::<Service>("io.containerd.cube.rs", None).await;
}

This pattern mirrors the actual entry point in CubeShim/shim/src/main.rs (lines 49-55).

Summary

• CubeShim implements the containerd shim v2 API in Rust, acting as a bridge between containerd and MicroVMs.
• The architecture uses ttrpc over vsock to communicate with the cube-agent inside the guest, forwarding container lifecycle commands.
• Core components are organized into modular directories: service/ for RPC handlers, hypervisor/ for VM management, and container/ for resource preparation.
• The Create and Start sequence involves hypervisor calls to cube_hypervisor::launch_vm followed by agent communication to execute processes.
• Registration requires setting runtime_type = "io.containerd.cube.v2" in containerd's configuration.

Frequently Asked Questions

What is the role of cube-agent in the CubeShim architecture?

The cube-agent is a process running inside the MicroVM that receives translated container commands from CubeShim via ttrpc. It acts as the final executor, spawning processes inside the guest and managing their I/O, effectively serving as the container runtime within the VM boundary.

How does CubeShim communicate with the MicroVM?

CubeShim communicates with the MicroVM using ttrpc (a lightweight RPC protocol) transported over vsock (virtual socket). This mechanism allows the host-side shim to send commands to the in-guest cube-agent without requiring network configuration, providing a secure, low-latency control channel.

What hypervisor does CubeShim use for VM management?

CubeShim integrates with CubeHypervisor, a RustVMM-based virtualization layer that uses KVM. The integration code in CubeShim/shim/src/hypervisor/cube_hypervisor.rs handles VM launch, snapshot, restore, and pause/resume operations, abstracting the underlying KVM interactions from the shim service layer.

Why does CubeShim implement the shim v2 API instead of v1?

The shim v2 specification provides a lightweight, long-running process model that isolates the container runtime from the containerd daemon. This allows CubeShim to manage MicroVM lifecycles independently, supporting features like auto-pause/resume and snapshotting without requiring containerd to maintain state or handle VM-specific complexity.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}