What Communication Protocol Is Used Between CubeShim and Cube-Agent?

CubeShim communicates with cube-agent via the containerd Shim v2 UpdateContainer RPC transported over ttrpc, using request annotations to dispatch specific actions.

In the TencentCloud/CubeSandbox repository, the communication protocol between CubeShim and cube-agent builds upon the standard containerd Shim v2 control-plane. Instead of implementing a custom transport layer, CubeShim extends the existing Shim v2 API to tunnel sandbox management commands through the UpdateContainer RPC method. This message travels over ttrpc, containerd’s lightweight RPC protocol designed specifically for shim-to-daemon communication.

The ttrpc Transport Layer

ttrpc (tiny transport RPC) is containerd’s lightweight protocol optimized for low-memory environments like shim processes. Unlike standard gRPC, ttrpc reduces memory overhead by eliminating the HTTP/2 framing requirement, making it ideal for the CubeShim-to-cube-agent communication path. The protocol operates over a Unix socket or pipe established by containerd when it spawns the shim process.

Because CubeShim implements the containerd Shim v2 interface, it automatically inherits this ttrpc transport mechanism. The shim exposes the standard Shim v2 service definitions, allowing CubeMaster to communicate with the sandbox without opening additional network ports or custom sockets.

UpdateContainer RPC as the Control Channel

The UpdateContainer RPC method—defined in the containerd Shim v2 API—serves as the primary control plane for CubeShim cube-agent communication. While standard containerd uses this RPC to update container resources, CubeSandbox repurposes it to transport arbitrary sandbox actions through the annotations field of the UpdateContainerRequest message.

Annotation-Based Action Dispatch

CubeShim interprets special annotation keys to determine which action to execute inside the sandbox. The primary dispatch key is cube.shimapi.update.action, which specifies the operation type (such as rollback or snapshot operations). Additional annotation keys carry operation-specific parameters.

According to the CubeShim/docs/shimapi/README.md documentation, this mechanism allows the extended control-plane to piggyback on the standard Shim v2 API without requiring custom RPC definitions.

Client-Side Implementation (CubeMaster)

When CubeMaster needs to trigger an action in the sandbox, it constructs an UpdateContainerRequest with the appropriate annotations and sends it via the ttrpc connection. The client code resides in the CubeMaster/.../shimapi package.

// Example: Cube API triggers a rollback action in the shim
req := &containerd.UpdateContainerRequest{
    ID: sandboxID,
    Annotations: map[string]string{
        "cube.shimapi.update.action": "RollbackSnapshot",
        "cube.shimapi.rollback.id":   snapshotID,
    },
}

// The request is sent over the ttrpc connection that containerd establishes
// with the shim process (see containerd/shim/v2 implementation).
client.UpdateContainer(ctx, req)

Server-Side Implementation (CubeShim)

Inside CubeShim, the UpdateContainer handler receives the request over ttrpc and dispatches to the appropriate internal handler based on the action annotation. This implementation is typically found in CubeShim/…/shim.go.

// Inside CubeShim – handling the incoming UpdateContainer RPC
func (s *Shim) UpdateContainer(ctx context.Context, req *containerd.UpdateContainerRequest) (*containerd.UpdateContainerResponse, error) {
    action := req.Annotations["cube.shimapi.update.action"]
    switch action {
    case "RollbackSnapshot":
        // call internal rollback handler
        s.handleRollback(req.Annotations["cube.shimapi.rollback.id"])
    // … other actions …
    }
    return &containerd.UpdateContainerResponse{}, nil
}

Step-by-Step Communication Flow

The complete CubeShim to cube-agent communication protocol follows this sequence:

1. Request Construction: CubeMaster builds an UpdateContainerRequest containing an annotations map with the cube.shimapi.update.action key and action-specific parameters.
2. ttrpc Transmission: The request travels over the existing ttrpc connection that containerd maintains with the CubeShim process.
3. RPC Receipt: CubeShim’s UpdateContainer method receives the request and extracts the action type from the annotations.
4. Dispatch: The shim dispatches to the appropriate handler (e.g., rollback handling documented in CubeShim/docs/shimapi/rollback-snapshot.md).
5. Execution: The handler interacts with cube-agent inside the sandbox VM to perform the requested operation.
6. Response: CubeShim returns an UpdateContainerResponse over the same ttrpc channel to acknowledge receipt.

Key Source Files

The implementation spans several locations in the TencentCloud/CubeSandbox repository:

• CubeShim/docs/shimapi/README.md: Documents the extended control-plane and the UpdateContainer RPC usage for shim-agent communication.
• CubeShim/…/shim.go: Contains the RPC server implementation that receives UpdateContainer calls over ttrpc and dispatches actions.
• CubeMaster/.../shimapi: Implements the client side that constructs UpdateContainerRequest messages with the required annotations.
• CubeShim/docs/shimapi/rollback-snapshot.md: Provides a concrete example of the RPC flow for the rollback action.

Summary

• CubeShim uses the containerd Shim v2 UpdateContainer RPC as the control channel for communicating with cube-agent.
• ttrpc serves as the underlying transport, leveraging containerd’s existing shim connection instead of custom network protocols.
• Annotations encode action types, with cube.shimapi.update.action serving as the dispatch key for operations like rollback and snapshot management.
• Implementation spans both CubeMaster and CubeShim, with the former constructing requests and the latter handling dispatch via the standard Shim v2 API.

Frequently Asked Questions

How does CubeShim maintain compatibility with standard containerd while extending the API?

CubeShim maintains full compatibility by implementing the standard containerd Shim v2 interface without modification. It repurposes the unused annotations field of the UpdateContainerRequest to carry Cube-specific commands. This approach ensures that standard containerd operations continue to work normally, while Cube-aware components can inject extended commands through the same RPC endpoint.

Why does CubeShim use annotations instead of dedicated RPC methods for sandbox actions?

Using annotations eliminates the need to modify the containerd Shim v2 protocol definitions or generate custom protobuf services. Since UpdateContainer already supports an arbitrary string map in its request structure, CubeShim leverages this existing field to tunnel action types and parameters. This design pattern allows CubeSandbox to extend functionality without forking containerd or maintaining a separate RPC infrastructure.

What specific actions can be triggered via the UpdateContainer RPC in CubeSandbox?

The primary documented action is RollbackSnapshot, which restores a sandbox to a previous state using an annotation like "cube.shimapi.update.action": "RollbackSnapshot" accompanied by "cube.shimapi.rollback.id": "<snapshot-id>". The architecture supports additional action types through the same dispatch mechanism, though specific implementations depend on the internal handlers registered in CubeShim/…/shim.go.

What is the difference between ttrpc and gRPC in this context?

ttrpc is a stripped-down RPC protocol designed by containerd specifically for shim processes, removing the HTTP/2 overhead required by standard gRPC. In the CubeSandbox communication flow, ttrpc reduces memory footprint and binary size for the CubeShim process while still providing the necessary request-response semantics for the UpdateContainer RPC. The wire format remains compatible with gRPC service definitions, but the transport layer uses a simpler framing mechanism optimized for local process communication.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}