How CubeShim Bridges containerd Shim v2 and the KVM Virtualization Layer

CubeShim acts as the integration layer between containerd’s Shim v2 runtime interface and KVM-based MicroVMs, translating high-level container operations into hardware virtualization commands through a Rust-based async service.

CubeShim is the core connectivity component in the TencentCloud/CubeSandbox repository that enables containerd to orchestrate KVM MicroVMs using standard container tooling. By implementing the containerd Shim v2 API, it allows Kubernetes and other CRI-compatible orchestrators to manage hardware-isolated workloads with the same commands used for traditional containers.

Architecture Overview

The integration follows a four-stage pipeline that moves from containerd RPC calls to executing raw KVM ioctls. Each stage is implemented in specific source modules that handle distinct responsibilities within the shim.

Shim Registration with containerd

CubeShim registers itself as a containerd Shim v2 runtime under the identifier io.containerd.cube.rs. In CubeShim/shim/src/main.rs, the binary entry point creates a Tokio async runtime and invokes shim_run::<Service>("io.containerd.cube.rs", ...), which establishes the ttrpc/gRPC endpoints that containerd uses to communicate with the shim.

When containerd receives a Create or Start request for a runtime configured with runtime_type = "io.containerd.cube.rs", it spawns the containerd-shim-cube-rs binary and forwards all subsequent sandbox operations to this service.

Sandbox Resource Preparation

The Service implementation in CubeShim/shim/src/service/mod.rs handles resource bundling before VM launch. This layer creates the sandbox root filesystem using CubeCoW (copy-on-write cloning), prepares memory backing files, and constructs the kernel command line parameters.

This preparation phase translates the OCI bundle format into a VmConfig structure that the underlying hypervisor can consume, bridging the gap between container specifications and virtualization requirements.

KVM Virtualization via Cube Hypervisor

CubeShim delegates actual virtualization to Cube Hypervisor, a thin Rust wrapper around RustVMM that issues raw KVM system calls. The critical path in CubeShim/shim/src/hypervisor/cube_hypervisor.rs (lines 75-135) follows this sequence:

• launch_vmm() (line 75): Creates a VmmInstance, loads the KVM kernel module, and establishes seccomp filters.
• create_vm(): Translates high-level configuration into RustVMM’s format and issues ioctl(VM_CREATE) and ioctl(VCPU_CREATE).
• boot_vm(): Triggers the final ioctl(VCPU_RUN) to start the MicroVM’s vCPU execution loop.

These calls map directly to Linux KVM syscalls, providing hardware-accelerated isolation without the overhead of legacy virtualization stacks.

Runtime Lifecycle Management

After boot, CubeShim maintains the connection between containerd and the running MicroVM. The shim opens a vsock channel to the guest for stdin/stdout/stderr forwarding and implements the shim-readiness handshake required by containerd v2.

For advanced operations like auto-pause, CubeShim supports in-place snapshotting via pause_vm_cube() and resume_vm_cube(), which serialize VM state to disk while preserving the hypervisor process for rapid restoration.

Step-by-Step Integration Flow

The complete execution path from ctr run to a running KVM MicroVM involves eight distinct phases:

1. Containerd invokes the shim: When users execute ctr run, containerd spawns containerd-shim-cube-rs, which parses arguments and initializes the async runtime in main.rs (lines 49-55).

2. Sandbox creation request: Cubelet (the node-local scheduler) sends a Create RPC to the shim’s Service implementation, triggering bundle preparation with CubeCoW cloning and memory file setup.

3. VMM initialization: The shim calls CubeHypervisor::launch_vmm() at line 75 in cube_hypervisor.rs, creating a VmmInstance with loaded KVM modules and seccomp whitelists.

4. VM configuration: The create_vm() method translates the shim’s VmConfig into RustVMM’s configuration format and sends ApiRequest::VmCreate to configure vCPUs, memory regions, and virtio devices.

5. MicroVM boot: boot_vm() sends a VmBoot request, causing RustVMM to execute the KVM ioctl(VCPU_RUN) loop and begin kernel execution.

6. I/O channel establishment: Once the guest kernel boots, the shim opens a vsock listener and begins forwarding stdio streams between containerd and the guest.

7. Lifecycle operations: When containerd requests pause/resume operations, the shim forwards these to pause_vm() or resume_vm(), with optimized paths using pause_vm_cube() for snapshot-based hibernation that writes state to a user-specified path.

8. Cleanup: On sandbox deletion, delete_vm() destroys the VM instance while optionally keeping the hypervisor process alive for future restores.

Implementation Examples

The following examples demonstrate how CubeShim integrates these layers in practice.

Building and Registering the Shim

To deploy CubeShim with containerd, build the binary and register it as a runtime:

# Build the shim binary (produces containerd-shim-cube-rs)

cd CubeShim && cargo build --release

# Configure containerd to use CubeShim

cat <<EOF > /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.rs"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube.options]
    Shim = "/usr/local/bin/containerd-shim-cube-rs"
EOF

# Create a sandbox (containerd invokes the shim automatically)

ctr run --rm -t docker.io/library/alpine:latest my-sandbox /bin/sh

This configuration tells containerd to use io.containerd.cube.rs for the cube runtime, invoking the containerd-shim-cube-rs binary for all sandbox operations.

Direct Hypervisor API Usage

For custom integrations, you can interact with the hypervisor layer directly:

use containerd_shim_cube_rs::service::Service;
use containerd_shim::Config;

// Initialize the shim service as done in main.rs
let cfg = Config { no_reaper: true, ..Default::default() };
let service = Service::new("sandbox-id".into());

// Launch the MicroVM through KVM
service.hypervisor.launch_vmm().await?;
service.hypervisor.create_vm(&vm_cfg).await?;
service.hypervisor.boot_vm().await?;

These calls map 1-to-1 to the methods defined in cube_hypervisor.rs, providing programmatic access to the KVM virtualization layer.

Snapshot and Restore Operations

CubeShim supports fast cold-start through snapshotting:

// Pause and snapshot to disk
service.hypervisor.pause_vm_cube("/tmp/snap.img").await?;

// Resume from snapshot later
service.hypervisor.resume_vm_cube("/tmp/snap.img").await?;

Under the hood, pause_vm_cube() uses VmPauseToSnapshot while resume_vm_cube() uses VmResumeFromSnapshot, enabling sub-second startup times for serverless workloads.

Key Source Files and Components

Understanding CubeShim requires familiarity with these critical files:

• CubeShim/shim/src/main.rs: Entry point that registers the shim with containerd and parses version flags. Implements the shim_run bootstrap logic and Tokio runtime initialization.

• CubeShim/shim/src/service/mod.rs: Core service implementation fulfilling the Shim v2 API contract, handling Create, Start, Pause, Resume, and Delete RPCs while preparing sandbox bundles.

• CubeShim/shim/src/hypervisor/cube_hypervisor.rs: Thin abstraction over RustVMM translating shim calls into KVM ioctls (lines 75-135 contain the critical launch sequence).

• docs/architecture/overview.md: Architecture documentation showing the data flow from Cubelet through CubeShim to Cube Hypervisor (lines 25-27).

• CubeShim/shim/src/common/mod.rs: Shared types like CResult and logging utilities used across the shim-hypervisor boundary.

Summary

• CubeShim implements the containerd_shim runtime interface under the identifier io.containerd.cube.rs, allowing containerd to orchestrate KVM MicroVMs using standard container tooling.
• The shim bridges to KVM through Cube Hypervisor, a RustVMM wrapper that issues direct ioctl calls for VM creation, vCPU setup, and virtio device configuration.
• Key integration points reside in main.rs (registration), service/mod.rs (resource preparation), and cube_hypervisor.rs (KVM virtualization).
• The architecture supports advanced features like in-place snapshots via pause_vm_cube() and resume_vm_cube(), enabling efficient serverless pause/resume cycles.
• All operations maintain the Shim v2 contract, providing seamless integration with existing containerd-based orchestrators like Kubernetes.

Frequently Asked Questions

How does CubeShim register with containerd?

CubeShim registers during binary startup in main.rs by calling shim_run::<Service>("io.containerd.cube.rs", ...), which creates the ttrpc/gRPC server endpoints. Containerd discovers the shim through the runtime_type configuration in /etc/containerd/config.toml, spawning containerd-shim-cube-rs whenever a sandbox requests the cube runtime.

What is the relationship between CubeShim and RustVMM?

CubeShim does not directly issue KVM ioctls. Instead, it delegates to Cube Hypervisor, a thin wrapper around RustVMM (the reference VMM written in Rust). The hypervisor translates high-level calls like create_vm() into sequences of ioctl(VM_CREATE), ioctl(VCPU_CREATE), and ioctl(VCPU_RUN) operations, as implemented in cube_hypervisor.rs.

How does CubeShim handle I/O between containerd and the MicroVM?

After the MicroVM boots, CubeShim establishes a vsock (virtual socket) connection to the guest. The shim forwards stdin, stdout, and stderr over this channel while also monitoring lifecycle events. This allows ctr commands and CRI clients to interact with the VM as if it were a standard container process.

Can CubeShim restore VMs from snapshots?

Yes. CubeShim implements pause_vm_cube() and resume_vm_cube() methods that use RustVMM's VmPauseToSnapshot and VmResumeFromSnapshot APIs. This enables the shim to write VM memory and state to a file (e.g., /tmp/snap.img) and later restore it, supporting fast cold-start patterns critical for serverless workloads.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}