# How CubeShim Bridges containerd Shim v2 and the KVM Virtualization Layer

> Learn how CubeShim connects containerd Shim v2 with KVM MicroVMs. This Rust-based service translates container operations into hardware virtualization commands for seamless integration.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-10

---

**CubeShim acts as the integration layer between containerd’s Shim v2 runtime interface and KVM-based MicroVMs, translating high-level container operations into hardware virtualization commands through a Rust-based async service.**

CubeShim is the core connectivity component in the TencentCloud/CubeSandbox repository that enables containerd to orchestrate KVM MicroVMs using standard container tooling. By implementing the containerd Shim v2 API, it allows Kubernetes and other CRI-compatible orchestrators to manage hardware-isolated workloads with the same commands used for traditional containers.

## Architecture Overview

The integration follows a four-stage pipeline that moves from containerd RPC calls to executing raw KVM ioctls. Each stage is implemented in specific source modules that handle distinct responsibilities within the shim.

### Shim Registration with containerd

CubeShim registers itself as a containerd Shim v2 runtime under the identifier [`io.containerd.cube.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/io.containerd.cube.rs). In [`CubeShim/shim/src/main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/main.rs), the binary entry point creates a Tokio async runtime and invokes `shim_run::<Service>("io.containerd.cube.rs", ...)`, which establishes the ttrpc/gRPC endpoints that containerd uses to communicate with the shim.

When containerd receives a `Create` or `Start` request for a runtime configured with `runtime_type = "io.containerd.cube.rs"`, it spawns the `containerd-shim-cube-rs` binary and forwards all subsequent sandbox operations to this service.

### Sandbox Resource Preparation

The `Service` implementation in [`CubeShim/shim/src/service/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/service/mod.rs) handles resource bundling before VM launch. This layer creates the sandbox root filesystem using **CubeCoW** (copy-on-write cloning), prepares memory backing files, and constructs the kernel command line parameters.

This preparation phase translates the OCI bundle format into a `VmConfig` structure that the underlying hypervisor can consume, bridging the gap between container specifications and virtualization requirements.

### KVM Virtualization via Cube Hypervisor

CubeShim delegates actual virtualization to **Cube Hypervisor**, a thin Rust wrapper around RustVMM that issues raw KVM system calls. The critical path in [`CubeShim/shim/src/hypervisor/cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/hypervisor/cube_hypervisor.rs) (lines 75-135) follows this sequence:

- **`launch_vmm()`** (line 75): Creates a `VmmInstance`, loads the KVM kernel module, and establishes seccomp filters.
- **`create_vm()`**: Translates high-level configuration into RustVMM’s format and issues `ioctl(VM_CREATE)` and `ioctl(VCPU_CREATE)`.
- **`boot_vm()`**: Triggers the final `ioctl(VCPU_RUN)` to start the MicroVM’s vCPU execution loop.

These calls map directly to Linux KVM syscalls, providing hardware-accelerated isolation without the overhead of legacy virtualization stacks.

### Runtime Lifecycle Management

After boot, CubeShim maintains the connection between containerd and the running MicroVM. The shim opens a vsock channel to the guest for stdin/stdout/stderr forwarding and implements the shim-readiness handshake required by containerd v2.

For advanced operations like auto-pause, CubeShim supports in-place snapshotting via `pause_vm_cube()` and `resume_vm_cube()`, which serialize VM state to disk while preserving the hypervisor process for rapid restoration.

## Step-by-Step Integration Flow

The complete execution path from `ctr run` to a running KVM MicroVM involves eight distinct phases:

1. **Containerd invokes the shim**: When users execute `ctr run`, containerd spawns `containerd-shim-cube-rs`, which parses arguments and initializes the async runtime in [`main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/main.rs) (lines 49-55).

2. **Sandbox creation request**: Cubelet (the node-local scheduler) sends a `Create` RPC to the shim’s `Service` implementation, triggering bundle preparation with CubeCoW cloning and memory file setup.

3. **VMM initialization**: The shim calls `CubeHypervisor::launch_vmm()` at line 75 in [`cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_hypervisor.rs), creating a `VmmInstance` with loaded KVM modules and seccomp whitelists.

4. **VM configuration**: The `create_vm()` method translates the shim’s `VmConfig` into RustVMM’s configuration format and sends `ApiRequest::VmCreate` to configure vCPUs, memory regions, and virtio devices.

5. **MicroVM boot**: `boot_vm()` sends a `VmBoot` request, causing RustVMM to execute the KVM `ioctl(VCPU_RUN)` loop and begin kernel execution.

6. **I/O channel establishment**: Once the guest kernel boots, the shim opens a vsock listener and begins forwarding stdio streams between containerd and the guest.

7. **Lifecycle operations**: When containerd requests pause/resume operations, the shim forwards these to `pause_vm()` or `resume_vm()`, with optimized paths using `pause_vm_cube()` for snapshot-based hibernation that writes state to a user-specified path.

8. **Cleanup**: On sandbox deletion, `delete_vm()` destroys the VM instance while optionally keeping the hypervisor process alive for future restores.

## Implementation Examples

The following examples demonstrate how CubeShim integrates these layers in practice.

### Building and Registering the Shim

To deploy CubeShim with containerd, build the binary and register it as a runtime:

```bash

# Build the shim binary (produces containerd-shim-cube-rs)

cd CubeShim && cargo build --release

# Configure containerd to use CubeShim

cat <<EOF > /etc/containerd/config.toml
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube]
  runtime_type = "io.containerd.cube.rs"
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.cube.options]
    Shim = "/usr/local/bin/containerd-shim-cube-rs"
EOF

# Create a sandbox (containerd invokes the shim automatically)

ctr run --rm -t docker.io/library/alpine:latest my-sandbox /bin/sh

```

This configuration tells containerd to use [`io.containerd.cube.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/io.containerd.cube.rs) for the `cube` runtime, invoking the `containerd-shim-cube-rs` binary for all sandbox operations.

### Direct Hypervisor API Usage

For custom integrations, you can interact with the hypervisor layer directly:

```rust
use containerd_shim_cube_rs::service::Service;
use containerd_shim::Config;

// Initialize the shim service as done in main.rs
let cfg = Config { no_reaper: true, ..Default::default() };
let service = Service::new("sandbox-id".into());

// Launch the MicroVM through KVM
service.hypervisor.launch_vmm().await?;
service.hypervisor.create_vm(&vm_cfg).await?;
service.hypervisor.boot_vm().await?;

```

These calls map 1-to-1 to the methods defined in [`cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_hypervisor.rs), providing programmatic access to the KVM virtualization layer.

### Snapshot and Restore Operations

CubeShim supports fast cold-start through snapshotting:

```rust
// Pause and snapshot to disk
service.hypervisor.pause_vm_cube("/tmp/snap.img").await?;

// Resume from snapshot later
service.hypervisor.resume_vm_cube("/tmp/snap.img").await?;

```

Under the hood, `pause_vm_cube()` uses `VmPauseToSnapshot` while `resume_vm_cube()` uses `VmResumeFromSnapshot`, enabling sub-second startup times for serverless workloads.

## Key Source Files and Components

Understanding CubeShim requires familiarity with these critical files:

- **[`CubeShim/shim/src/main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/main.rs)**: Entry point that registers the shim with containerd and parses version flags. Implements the `shim_run` bootstrap logic and Tokio runtime initialization.

- **[`CubeShim/shim/src/service/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/service/mod.rs)**: Core service implementation fulfilling the Shim v2 API contract, handling `Create`, `Start`, `Pause`, `Resume`, and `Delete` RPCs while preparing sandbox bundles.

- **[`CubeShim/shim/src/hypervisor/cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/hypervisor/cube_hypervisor.rs)**: Thin abstraction over RustVMM translating shim calls into KVM ioctls (lines 75-135 contain the critical launch sequence).

- **[`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md)**: Architecture documentation showing the data flow from Cubelet through CubeShim to Cube Hypervisor (lines 25-27).

- **[`CubeShim/shim/src/common/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/common/mod.rs)**: Shared types like `CResult` and logging utilities used across the shim-hypervisor boundary.

## Summary

- CubeShim implements the `containerd_shim` runtime interface under the identifier [`io.containerd.cube.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/io.containerd.cube.rs), allowing containerd to orchestrate KVM MicroVMs using standard container tooling.
- The shim bridges to KVM through **Cube Hypervisor**, a RustVMM wrapper that issues direct `ioctl` calls for VM creation, vCPU setup, and virtio device configuration.
- Key integration points reside in [`main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/main.rs) (registration), [`service/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/service/mod.rs) (resource preparation), and [`cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_hypervisor.rs) (KVM virtualization).
- The architecture supports advanced features like in-place snapshots via `pause_vm_cube()` and `resume_vm_cube()`, enabling efficient serverless pause/resume cycles.
- All operations maintain the Shim v2 contract, providing seamless integration with existing containerd-based orchestrators like Kubernetes.

## Frequently Asked Questions

### How does CubeShim register with containerd?

CubeShim registers during binary startup in [`main.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/main.rs) by calling `shim_run::<Service>("io.containerd.cube.rs", ...)`, which creates the ttrpc/gRPC server endpoints. Containerd discovers the shim through the `runtime_type` configuration in [`/etc/containerd/config.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main//etc/containerd/config.toml), spawning `containerd-shim-cube-rs` whenever a sandbox requests the cube runtime.

### What is the relationship between CubeShim and RustVMM?

CubeShim does not directly issue KVM ioctls. Instead, it delegates to **Cube Hypervisor**, a thin wrapper around RustVMM (the reference VMM written in Rust). The hypervisor translates high-level calls like `create_vm()` into sequences of `ioctl(VM_CREATE)`, `ioctl(VCPU_CREATE)`, and `ioctl(VCPU_RUN)` operations, as implemented in [`cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_hypervisor.rs).

### How does CubeShim handle I/O between containerd and the MicroVM?

After the MicroVM boots, CubeShim establishes a vsock (virtual socket) connection to the guest. The shim forwards stdin, stdout, and stderr over this channel while also monitoring lifecycle events. This allows `ctr` commands and CRI clients to interact with the VM as if it were a standard container process.

### Can CubeShim restore VMs from snapshots?

Yes. CubeShim implements `pause_vm_cube()` and `resume_vm_cube()` methods that use RustVMM's `VmPauseToSnapshot` and `VmResumeFromSnapshot` APIs. This enables the shim to write VM memory and state to a file (e.g., `/tmp/snap.img`) and later restore it, supporting fast cold-start patterns critical for serverless workloads.