# What Communication Protocol Is Used Between CubeShim and Cube-Agent?

> CubeShim and cube-agent communicate using ttrpc, leveraging containerd Shim v2 UpdateContainer RPC and request annotations for efficient action dispatching.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-05

---

**CubeShim communicates with cube-agent via the containerd Shim v2 `UpdateContainer` RPC transported over **ttrpc**, using request annotations to dispatch specific actions.**

In the TencentCloud/CubeSandbox repository, the **communication protocol between CubeShim and cube-agent** builds upon the standard containerd Shim v2 control-plane. Instead of implementing a custom transport layer, CubeShim extends the existing Shim v2 API to tunnel sandbox management commands through the `UpdateContainer` RPC method. This message travels over **ttrpc**, containerd’s lightweight RPC protocol designed specifically for shim-to-daemon communication.

## The ttrpc Transport Layer

**ttrpc** (tiny transport RPC) is containerd’s lightweight protocol optimized for low-memory environments like shim processes. Unlike standard gRPC, ttrpc reduces memory overhead by eliminating the HTTP/2 framing requirement, making it ideal for the CubeShim-to-cube-agent communication path. The protocol operates over a Unix socket or pipe established by containerd when it spawns the shim process.

Because CubeShim implements the containerd Shim v2 interface, it automatically inherits this ttrpc transport mechanism. The shim exposes the standard Shim v2 service definitions, allowing CubeMaster to communicate with the sandbox without opening additional network ports or custom sockets.

## UpdateContainer RPC as the Control Channel

The `UpdateContainer` RPC method—defined in the containerd Shim v2 API—serves as the primary control plane for **CubeShim cube-agent communication**. While standard containerd uses this RPC to update container resources, CubeSandbox repurposes it to transport arbitrary sandbox actions through the `annotations` field of the `UpdateContainerRequest` message.

### Annotation-Based Action Dispatch

CubeShim interprets special annotation keys to determine which action to execute inside the sandbox. The primary dispatch key is `cube.shimapi.update.action`, which specifies the operation type (such as rollback or snapshot operations). Additional annotation keys carry operation-specific parameters.

According to the [`CubeShim/docs/shimapi/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/docs/shimapi/README.md) documentation, this mechanism allows the extended control-plane to piggyback on the standard Shim v2 API without requiring custom RPC definitions.

### Client-Side Implementation (CubeMaster)

When CubeMaster needs to trigger an action in the sandbox, it constructs an `UpdateContainerRequest` with the appropriate annotations and sends it via the ttrpc connection. The client code resides in the `CubeMaster/.../shimapi` package.

```go
// Example: Cube API triggers a rollback action in the shim
req := &containerd.UpdateContainerRequest{
    ID: sandboxID,
    Annotations: map[string]string{
        "cube.shimapi.update.action": "RollbackSnapshot",
        "cube.shimapi.rollback.id":   snapshotID,
    },
}

// The request is sent over the ttrpc connection that containerd establishes
// with the shim process (see containerd/shim/v2 implementation).
client.UpdateContainer(ctx, req)

```

### Server-Side Implementation (CubeShim)

Inside CubeShim, the `UpdateContainer` handler receives the request over ttrpc and dispatches to the appropriate internal handler based on the action annotation. This implementation is typically found in `CubeShim/…/shim.go`.

```go
// Inside CubeShim – handling the incoming UpdateContainer RPC
func (s *Shim) UpdateContainer(ctx context.Context, req *containerd.UpdateContainerRequest) (*containerd.UpdateContainerResponse, error) {
    action := req.Annotations["cube.shimapi.update.action"]
    switch action {
    case "RollbackSnapshot":
        // call internal rollback handler
        s.handleRollback(req.Annotations["cube.shimapi.rollback.id"])
    // … other actions …
    }
    return &containerd.UpdateContainerResponse{}, nil
}

```

## Step-by-Step Communication Flow

The complete **CubeShim to cube-agent communication protocol** follows this sequence:

1. **Request Construction**: CubeMaster builds an `UpdateContainerRequest` containing an `annotations` map with the `cube.shimapi.update.action` key and action-specific parameters.
2. **ttrpc Transmission**: The request travels over the existing ttrpc connection that containerd maintains with the CubeShim process.
3. **RPC Receipt**: CubeShim’s `UpdateContainer` method receives the request and extracts the action type from the annotations.
4. **Dispatch**: The shim dispatches to the appropriate handler (e.g., rollback handling documented in [`CubeShim/docs/shimapi/rollback-snapshot.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/docs/shimapi/rollback-snapshot.md)).
5. **Execution**: The handler interacts with cube-agent inside the sandbox VM to perform the requested operation.
6. **Response**: CubeShim returns an `UpdateContainerResponse` over the same ttrpc channel to acknowledge receipt.

## Key Source Files

The implementation spans several locations in the TencentCloud/CubeSandbox repository:

- **[`CubeShim/docs/shimapi/README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/docs/shimapi/README.md)**: Documents the extended control-plane and the `UpdateContainer` RPC usage for shim-agent communication.
- **`CubeShim/…/shim.go`**: Contains the RPC server implementation that receives `UpdateContainer` calls over ttrpc and dispatches actions.
- **`CubeMaster/.../shimapi`**: Implements the client side that constructs `UpdateContainerRequest` messages with the required annotations.
- **[`CubeShim/docs/shimapi/rollback-snapshot.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/docs/shimapi/rollback-snapshot.md)**: Provides a concrete example of the RPC flow for the rollback action.

## Summary

- **CubeShim uses the containerd Shim v2 `UpdateContainer` RPC** as the control channel for communicating with cube-agent.
- **ttrpc serves as the underlying transport**, leveraging containerd’s existing shim connection instead of custom network protocols.
- **Annotations encode action types**, with `cube.shimapi.update.action` serving as the dispatch key for operations like rollback and snapshot management.
- **Implementation spans both CubeMaster and CubeShim**, with the former constructing requests and the latter handling dispatch via the standard Shim v2 API.

## Frequently Asked Questions

### How does CubeShim maintain compatibility with standard containerd while extending the API?

CubeShim maintains full compatibility by implementing the standard containerd Shim v2 interface without modification. It repurposes the unused `annotations` field of the `UpdateContainerRequest` to carry Cube-specific commands. This approach ensures that standard containerd operations continue to work normally, while Cube-aware components can inject extended commands through the same RPC endpoint.

### Why does CubeShim use annotations instead of dedicated RPC methods for sandbox actions?

Using annotations eliminates the need to modify the containerd Shim v2 protocol definitions or generate custom protobuf services. Since `UpdateContainer` already supports an arbitrary string map in its request structure, CubeShim leverages this existing field to tunnel action types and parameters. This design pattern allows CubeSandbox to extend functionality without forking containerd or maintaining a separate RPC infrastructure.

### What specific actions can be triggered via the UpdateContainer RPC in CubeSandbox?

The primary documented action is **RollbackSnapshot**, which restores a sandbox to a previous state using an annotation like `"cube.shimapi.update.action": "RollbackSnapshot"` accompanied by `"cube.shimapi.rollback.id": "<snapshot-id>"`. The architecture supports additional action types through the same dispatch mechanism, though specific implementations depend on the internal handlers registered in `CubeShim/…/shim.go`.

### What is the difference between ttrpc and gRPC in this context?

**ttrpc** is a stripped-down RPC protocol designed by containerd specifically for shim processes, removing the HTTP/2 overhead required by standard **gRPC**. In the CubeSandbox communication flow, ttrpc reduces memory footprint and binary size for the CubeShim process while still providing the necessary request-response semantics for the `UpdateContainer` RPC. The wire format remains compatible with gRPC service definitions, but the transport layer uses a simpler framing mechanism optimized for local process communication.