How CubeShim Integrates with containerd's Shim v2 API for Sandbox Lifecycle Management

CubeShim implements containerd's Shim v2 API by exposing a ttrpc server that translates Shim v2 RPCs—such as CreateSandbox, StartSandbox, and DeleteSandbox—into VM operations via Cubelet, enabling containerd to manage sandbox lifecycles across hypervisor boundaries.

CubeShim (binary containerd-shim-cube-rs) serves as the bridge between containerd's generic runtime interface and CubeSandbox's VM-based execution environment. Located in the TencentCloud/CubeSandbox repository, this Rust-based shim implements the containerd Shim v2 protocol to orchestrate sandbox creation, execution, and teardown through the Cubelet component.

Shim v2 Entry Point and ttrpc Server Architecture

When containerd initiates a sandbox operation, it spawns the containerd-shim-cube-rs binary as a child process. The shim initializes a ttrpc server that implements the protobuf interfaces defined by containerd-shim-protos.

In CubeShim/shim/src/main.rs, the entry point registers the generated service handlers:

// CubeShim entry point – starts the ttrpc server
fn main() -> Result<()> {
    let server = ttrpc::server::Builder::new()
        .register_service(oci::create_sandbox_server::<ShimService>())
        .register_service(oci::start_sandbox_server::<ShimService>())
        .register_service(oci::delete_sandbox_server::<ShimService>())
        .build();
    server.listen_and_serve("/run/containerd/containerd.sock")?;
    Ok(())
}

The server binds to a Unix domain socket and listens for incoming RPC requests from containerd, effectively acting as the Shim v2 endpoint.

Implementing the Shim v2 API Surface

CubeShim implements the standard Shim v2 API surface defined in CubeShim/protoc/src/oci.proto and CubeShim/protoc/src/health.proto. These protobuf specifications declare the service methods that containerd expects, including CreateSandbox, StartSandbox, DeleteSandbox, State, Exec, and HealthCheck.

The concrete implementation resides in CubeShim/shim/src/service/mod.rs and CubeShim/shim/src/service/srv.rs. The ShimService struct defined in these files provides the logic for each RPC, maintaining a local state machine protected by a mutex to track sandbox transitions.

Sandbox Lifecycle Delegation to Cubelet

Each Shim v2 request maps to a higher-level operation on the Cubelet side. The shim acts as a client to Cubelet's ttrpc service, forwarding requests to the VM manager:

• CreateSandbox – The shim constructs an OCI bundle from the request, then invokes the Cubelet client to launch a VM via the hypervisor. This is implemented in the hypervisor interaction layer at CubeShim/shim/src/hypervisor/cube_hypervisor.rs.

• StartSandbox – The shim instructs Cubelet to start the VM and establishes a vsock connection for control and I/O channel initialization.

• DeleteSandbox – The shim signals Cubelet to shut down the VM, detach resources, and clean up the bundle directory.

• Exec/State – These operations forward execution requests and state queries through the same ttrpc channel to the running VM.

The communication between shim and Cubelet uses ttrpc over Unix domain sockets, while data plane traffic (stdout/stderr, exec I/O) flows through dedicated vsock channels.

The following example shows how CreateSandbox forwards the request to Cubelet:

// Example implementation of CreateSandbox – forwards to Cubelet
impl ShimService {
    async fn create_sandbox(
        &self,
        req: Request<CreateSandboxRequest>,
    ) -> Result<Response<CreateSandboxResponse>, ttrpc::Error> {
        // Build OCI bundle, then invoke Cubelet’s sandbox creation RPC
        let cubelet_client = CubeletClient::new("/run/cubelet.sock");
        let resp = cubelet_client
            .create_sandbox(req.into_inner())
            .await?;
        Ok(Response::new(resp))
    }
}

For exec operations, the shim opens a vsock stream to pipe I/O between the ttrpc client and the VM:

// Forwarding exec I/O over vsock (simplified)
async fn exec(
    &self,
    req: Request<ExecRequest>,
) -> Result<Response<ExecResponse>, ttrpc::Error> {
    // Open a vsock stream to the VM (fd 0 = control, fd 1 = stdio)
    let vsock = VsockConn::connect(self.vm_id, VM_EXEC_PORT).await?;
    // Pipe stdin/stdout between the ttrpc client and the vsock
    let (tx, rx) = tokio::io::split(vsock);
    // ...
    Ok(Response::new(exec_resp))
}

State Management and Health Monitoring

The shim maintains an authoritative view of the sandbox state by subscribing to hypervisor events (VM crash, pause, resume) and updating containerd via the Shim v2 API. This synchronization ensures that containerd's state machine remains consistent with the actual VM status.

Health checks are implemented according to health.proto, allowing containerd to verify shim responsiveness before issuing critical lifecycle commands.

Logging and I/O Forwarding

CubeShim injects the annotation cube.container.log_forwarding=true into the OCI spec during sandbox creation. The Cube Agent inside the VM creates buffered pipes for the init process and streams logs over a dedicated vsock. The shim receives these streams and writes them to host-side log files located under /data/log/CubeShim/.

Key Implementation Files

The following source files define the integration between CubeShim and containerd's Shim v2 API:

• CubeShim/shim/src/main.rs – Entry point that initializes the ttrpc server and registers Shim v2 services.
• CubeShim/shim/src/service/mod.rs – Core ShimService struct implementing the Shim v2 RPC handlers.
• CubeShim/shim/src/service/srv.rs – Glue code wiring protobuf-generated servers to the ShimService implementation.
• CubeShim/shim/src/hypervisor/cube_hypervisor.rs – Hypervisor interface for VM launch, pause, resume, and destruction.
• CubeShim/protoc/src/oci.proto – Containerd Shim v2 service definitions used by the shim.
• CubeShim/protoc/src/health.proto – Health-check service definitions for shim monitoring.
• CubeShim/docs/shimapi/README.md – Documentation covering the Shim v2 API surface provided by CubeShim.

Summary

• CubeShim (containerd-shim-cube-rs) implements the containerd Shim v2 API by exposing a ttrpc server that handles lifecycle RPCs.
• It translates CreateSandbox, StartSandbox, and DeleteSandbox calls into VM operations via the Cubelet component.
• The shim communicates with Cubelet using ttrpc over Unix sockets and uses vsock for data plane I/O forwarding.
• State management is synchronized between the hypervisor, Cubelet, and containerd through a protected state machine.
• Logging is facilitated by injecting OCI annotations and streaming logs from the VM to host-side files.

Frequently Asked Questions

What is the role of ttrpc in CubeShim's architecture?

ttrpc provides the lightweight RPC transport between containerd and CubeShim, as well as between CubeShim and Cubelet. The shim registers a ttrpc server in main.rs to accept Shim v2 requests from containerd, and uses a ttrpc client to forward these requests to Cubelet for VM management.

How does CubeShim handle sandbox deletion?

When containerd calls DeleteSandbox, the shim implementation in CubeShim/shim/src/service/mod.rs sends a shutdown signal to Cubelet via ttrpc. Cubelet then instructs the hypervisor (through cube_hypervisor.rs) to destroy the VM, release resources, and remove the OCI bundle directory.

Where are the Shim v2 API definitions located in the CubeSandbox repository?

The protobuf definitions for the Shim v2 API are located in CubeShim/protoc/src/oci.proto and CubeShim/protoc/src/health.proto. These files define the service methods and message types that CubeShim implements to satisfy the containerd Shim v2 contract.

How does CubeShim forward container logs to the host?

CubeShim injects the annotation cube.container.log_forwarding=true into the OCI spec. The Cube Agent inside the VM streams logs over a vsock connection, which the shim receives and writes to host-side log files under /data/log/CubeShim/, ensuring persistent log storage outside the VM.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}