How CubeShim Integrates with containerd Shim v2 API for Sandbox Lifecycle Management

CubeShim implements the containerd Shim v2 API via a ttrpc server that translates sandbox management RPCs into VM lifecycle operations, delegating hypervisor calls to Cubelet while maintaining state synchronization with containerd.

TencentCloud/CubeSandbox uses CubeShim (binary name containerd-shim-cube-rs) as the bridge between containerd's generic runtime interface and its VM-based sandbox runtime. By implementing the Shim v2 protobuf contracts defined in containerd-shim-protos, CubeShim enables containerd to manage micro-VM lifecycles through standard container operations while abstracting hypervisor-specific logic behind the Shim v2 API surface.

Shim v2 API Architecture and Entry Point

When containerd creates a sandbox, it spawns the containerd-shim-cube-rs binary as a child process. In CubeShim/shim/src/main.rs, the shim initializes a ttrpc server and registers the service handlers that implement the Shim v2 interface described in CubeShim/docs/shimapi/README.md. This server listens on a Unix domain socket and exposes the standard lifecycle methods required by containerd's runtime v2 architecture.

The protobuf definitions in CubeShim/protoc/src/oci.proto and CubeShim/protoc/src/health.proto define the service contracts, including CreateSandbox, StartSandbox, DeleteSandbox, State, and Exec. These definitions are compiled into Rust code and implemented by the ShimService struct in CubeShim/shim/src/service/mod.rs.

RPC Service Implementation

The service layer wires the generated protobuf servers to the business logic. CubeShim/shim/src/service/srv.rs registers the ttrpc services, while CubeShim/shim/src/service/mod.rs contains the core ShimService implementation that handles incoming RPCs.

Each sandbox operation translates to a specific sequence of actions:

• CreateSandbox – Validates the OCI bundle, prepares the root filesystem, and initiates the VM creation sequence.
• StartSandbox – Boots the micro-VM through the hypervisor interface.
• DeleteSandbox – Signals the hypervisor to destroy the VM and cleans up the bundle directory.
• Exec and State – Forwards process execution requests and state queries to the running sandbox.

Sandbox Lifecycle Management

CreateSandbox and Bundle Preparation

When containerd calls CreateSandbox, the shim receives a CreateSandboxRequest containing the OCI specification and sandbox configuration. The implementation in CubeShim/shim/src/service/mod.rs constructs the bundle directory and invokes the Cubelet client via ttrpc to allocate resources. The shim stores the sandbox state locally in a state machine protected by synchronization primitives.

StartSandbox and VM Initialization

The StartSandbox RPC triggers the actual VM boot sequence. The shim delegates to CubeShim/shim/src/hypervisor/cube_hypervisor.rs, which interfaces with the underlying hypervisor (e.g., QEMU, Cloud Hypervisor) to launch the micro-VM. During this phase, the shim establishes vsock connections for control plane and data plane communication between the host and the guest agent.

DeleteSandbox and Resource Cleanup

DeleteSandbox initiates a graceful shutdown of the VM. The shim sends a termination signal through the hypervisor interface, waits for the process to exit, and then removes the sandbox directory and associated vsock endpoints. This ensures containerd’s view of resource availability remains consistent with the actual system state.

Exec and State Operations

For process execution within an existing sandbox, the shim opens a vsock stream to the VM's agent and pipes stdin/stdout between the ttrpc client and the vsock connection. State queries return the cached status maintained by the shim's state machine, which subscribes to hypervisor events for real-time updates.

Communication with Cubelet

CubeShim communicates with Cubelet (the node-level sandbox manager) over ttrpc via Unix domain sockets. While the Shim v2 API faces containerd, the shim acts as a client to Cubelet's internal API, forwarding create, start, and delete requests.

Data plane traffic—such as container logs and exec I/O—travels over vsock channels between the shim and the agent running inside the micro-VM. This separation of control (ttrpc) and data (vsock) paths ensures that heavy I/O does not block RPC management traffic.

Implementation Examples

The following patterns illustrate the integration between the Shim v2 API surface and CubeSandbox internals.

Initialize the ttrpc server and register Shim v2 services in the main entry point:

// CubeShim/shim/src/main.rs
fn main() -> Result<()> {
    let server = ttrpc::server::Builder::new()
        .register_service(oci::create_sandbox_server::<ShimService>())
        .register_service(oci::start_sandbox_server::<ShimService>())
        .register_service(oci::delete_sandbox_server::<ShimService>())
        .build();
    server.listen_and_serve("/run/containerd/containerd.sock")?;
    Ok(())
}

Forward CreateSandbox requests to Cubelet after preparing the OCI bundle:

// CubeShim/shim/src/service/mod.rs
impl ShimService {
    async fn create_sandbox(
        &self,
        req: Request<CreateSandboxRequest>,
    ) -> Result<Response<CreateSandboxResponse>, ttrpc::Error> {
        // Prepare the bundle directory from the OCI spec
        let bundle = Bundle::from_request(&req)?;
        
        // Delegate VM creation to Cubelet via ttrpc
        let cubelet_client = CubeletClient::new("/run/cubelet.sock");
        let resp = cubelet_client
            .create_sandbox(req.into_inner())
            .await?;
            
        // Update local state machine
        self.state_manager.insert(resp.sandbox_id, State::Created);
        Ok(Response::new(resp))
    }
}

Stream exec I/O over vsock between the containerd client and the VM agent:

// CubeShim/shim/src/service/mod.rs
async fn exec(
    &self,
    req: Request<ExecRequest>,
) -> Result<Response<ExecResponse>, ttrpc::Error> {
    let vm_id = req.get_ref().sandbox_id;
    
    // Establish vsock connection to the VM agent
    let vsock = VsockConn::connect(vm_id, VM_EXEC_PORT).await?;
    let (mut read_half, mut write_half) = tokio::io::split(vsock);
    
    // Pipe streams between ttrpc client and vsock
    tokio::spawn(async move {
        tokio::io::copy(&mut read_half, &mut stdout).await.ok();
    });
    
    Ok(Response::new(ExecResponse::default()))
}

Summary

• CubeShim (containerd-shim-cube-rs) implements the containerd Shim v2 API as a ttrpc server defined in CubeShim/protoc/src/oci.proto.
• Lifecycle RPCs (CreateSandbox, StartSandbox, DeleteSandbox) are handled in CubeShim/shim/src/service/mod.rs and delegated to Cubelet via ttrpc.
• The shim manages VM state through CubeShim/shim/src/hypervisor/cube_hypervisor.rs, interacting with the hypervisor to boot and destroy micro-VMs.
• vsock channels provide data plane connectivity for exec I/O and logging, while ttrpc handles control plane communication.
• Service registration and RPC routing logic resides in CubeShim/shim/src/service/srv.rs.

Frequently Asked Questions

What role does ttrpc play in CubeShim's architecture?

ttrpc provides the RPC transport for both the Shim v2 API (facing containerd) and the internal Cubelet API (facing the node agent). It offers lower latency and smaller footprint compared to gRPC, which is critical for shim processes that must remain resident for the duration of the sandbox lifecycle.

How does CubeShim maintain state consistency with containerd?

The shim maintains a local state machine in CubeShim/shim/src/service/mod.rs that tracks sandbox transitions (Creating, Running, Stopped). It subscribes to hypervisor events via cube_hypervisor.rs to detect VM crashes or exits, then updates containerd through the Shim v2 State RPC to ensure the orchestrator's view matches the actual VM status.

What is the difference between CubeShim and Cubelet?

CubeShim (containerd-shim-cube-rs) is the per-sandbox process implementing containerd's Shim v2 API, responsible for RPC handling and containerd integration. Cubelet is the node-level daemon that manages global resources, handles hypervisor interactions on behalf of multiple shims, and coordinates VM placement.

How are exec processes handled across the VM boundary?

When containerd requests an exec via the Shim v2 Exec RPC, CubeShim opens a vsock connection to the agent running inside the target micro-VM. The shim then proxies stdin, stdout, and stderr between the containerd client stream and the vsocket, allowing interactive commands to execute inside the sandbox while the shim process remains on the host.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}