# How ttrpc Protocol Over vsock Enables Shim-to-Agent Communication in CubeSandbox

> Discover how CubeSandbox uses ttrpc over vsock for efficient shim-to-agent communication. Learn about low-latency, kernel-mediated control channels in this technical deep dive.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: internals
- Published: 2026-07-03

---

**CubeSandbox utilizes ttrpc (tiny RPC) over vsock (virtual socket) to create a low-latency, kernel-mediated control channel between the containerd shim inside the VM and the Cubelet agent on the host, eliminating the need for IP networking while maintaining efficient container lifecycle management.**

CubeSandbox, available in the TencentCloud/CubeSandbox repository, implements a specialized communication mechanism where the containerd shim running inside each sandboxed VM establishes a direct control channel with the host-side Cubelet service. This architecture leverages the ttrpc protocol over vsock to handle container lifecycle operations—such as pause, resume, and state queries—without requiring traditional network stacks or IP configuration overhead.

## Architecture Overview

The shim-to-agent communication stack consists of two distinct layers working in concert:

- **vsock (virtual socket)** – Provides transport-layer connectivity between the VM (CID 2) and the host (CID 0) without an IP stack, using the kernel-mediated `AF_VSOCK` address family.
- **ttrpc (tiny RPC)** – A lightweight, protobuf-based RPC protocol originally designed for containerd shims, instantiated on top of the vsock connection to provide request-response semantics.

### The Hybrid VSock Abstraction Layer

Both the shim and the agent import the `Cubelet/plugins/chi/vsockets` package to handle connection establishment. This package defines a **HybridVSock** URL format (`hvsock://<cid>:<port>`) that abstracts the underlying transport, allowing the code to parse vsock connections for production use or fall back to plain TCP connections for debugging purposes.

In [`Cubelet/plugins/chi/vsockets/vsockets.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/vsockets.go), the `HybridVSock` struct and `HybridVSockDialer` helper handle the parsing logic and connection establishment. The dialer accepts a timeout parameter and returns a net.Conn-compatible interface that works with the ttrpc client constructors.

### Server-Side Implementation in Cubelet

When the Cubelet (agent) starts, it creates a vsock listener and attaches a ttrpc server to handle incoming shim requests. In [`Cubelet/cmd/cubelet/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/cmd/cubelet/main.go), the code constructs the ttrpc address by appending the `.ttrpc` suffix to the configured gRPC address, enabling the shim to discover the correct vsock port.

The server implementation in [`Cubelet/plugins/chi/vsockets/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/server.go) follows this pattern:

```go
// Cubelet/plugins/chi/vsockets/server.go
l, err := vsock.Listen(uint32(port), &vsock.Config{})
if err != nil {
    return nil, fmt.Errorf("failed to listen vsock: %w", err)
}
ttrpcSrv := ttrpc.NewServer()
ttrpcSrv.Register(... )          // register the containerd shim services
go ttrpcSrv.Serve(l)             // runs until the listener is closed

```

The listener operates on the host side (CID 0) and accepts connections from the VM shim (CID 2), forwarding them to the registered ttrpc service handlers.

### Client-Side Implementation in the Shim

Inside the VM, the shim obtains a vsock connection via the HybridVSock dialer and constructs a ttrpc client. The implementation in [`Cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/update.go) demonstrates this pattern:

```go
// Cubelet/services/cubebox/update.go (excerpt)
conn, err := vsockets.HybridVSockDialer(hvsockURL, timeout)
if err != nil {
    return nil, err
}
ttrpcClient := ttrpc.NewClient(conn, &ttrpc.ClientOptions{})
defer ttrpcClient.Close()

```

The shim then invokes RPC methods such as `Pause`, `Resume`, or `State` on the agent. Context metadata for request tracing is transferred using `ttrpc.WithMetadata` and retrieved with `ttrpc.GetMetadata`, as implemented in the same file.

## Performance and Security Characteristics

The combination of vsock and ttrpc provides specific advantages for sandboxed container environments:

| Aspect | vsock Benefit | ttrpc Benefit |
|--------|--------------|---------------|
| **Network Independence** | Works directly between VM and host; no IP configuration or bridge interfaces required. | Designed for low-overhead protobuf RPC; minimal framing overhead compared to HTTP/2. |
| **Latency** | Kernel-mediated, low-latency path with no packet-loss overhead or network stack traversal. | Binary-encoded messages eliminate text parsing overhead; smaller header sizes than gRPC. |
| **Security** | VSock is isolated to the VM-host pair (CID filtering); traffic never traverses physical networks. | Protocol transmits only defined protobuf messages; metadata can carry authentication tokens if needed. |

This design eliminates the need for a full network stack inside the VM while retaining the rich RPC semantics required for container lifecycle management.

## Implementation Examples

### Establishing a vsock Connection from the Shim

To connect the shim to the Cubelet agent, use the HybridVSock abstraction layer with a vsock URL specifying CID 2 (VM) and the target port:

```go
import (
    "time"
    "github.com/TencentCloud/CubeSandbox/Cubelet/plugins/chi/vsockets"
    "github.com/containerd/ttrpc"
)

// Build a URL like "hvsock://2:12345"
hvsockURL := vsockets.NewHybridVSock(2, 12345).String()

// Connect with a 5-second timeout
conn, err := vsockets.HybridVSockDialer(hvsockURL, 5*time.Second)
if err != nil {
    // handle error
}

// Create a ttrpc client on the vsock connection
client := ttrpc.NewClient(conn, &ttrpc.ClientOptions{})
defer client.Close()

```

*Reference*: [`Cubelet/plugins/chi/vsockets/vsockets.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/vsockets.go) and [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go)

### Configuring the ttrpc Server on vsock

The Cubelet agent listens on a vsock endpoint and registers containerd shim services:

```go
import (
    "fmt"
    "github.com/mdlayher/vsock"
    "github.com/containerd/ttrpc"
    "github.com/TencentCloud/CubeSandbox/Cubelet/internal/shimapi"
)

func serveTTRPC(port uint32) error {
    // Listen on host CID (0) for connections from VM CID (2)
    listener, err := vsock.Listen(port, &vsock.Config{})
    if err != nil {
        return fmt.Errorf("vsock listen failed: %w", err)
    }

    srv := ttrpc.NewServer()
    // Register the containerd shim RPC interfaces
    shimapi.RegisterTaskService(srv, taskService)

    // Serve until the listener is closed
    go srv.Serve(listener)
    return nil
}

```

*Reference*: [`Cubelet/plugins/chi/vsockets/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/server.go)

### Handling RPC Metadata and Context

The shim can attach request metadata (such as trace IDs) before calling agent methods:

```go
import (
    "context"
    "github.com/containerd/ttrpc"
)

// Attach metadata (e.g., request ID) before a call
md := ttrpc.MD{"request-id": []string{"42"}}
ctx := ttrpc.WithMetadata(context.Background(), md)

// Remote call
resp, err := client.TaskService().Pause(ctx, &taskapi.PauseRequest{...})

// Retrieve metadata from the response context if needed
outMD, _ := ttrpc.GetMetadata(ctx)

```

*Reference*: [`Cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/update.go)

## Summary

- CubeSandbox implements **ttrpc over vsock** to provide a lightweight, secure control channel between the VM shim and the host Cubelet agent.
- The **HybridVSock** abstraction in [`Cubelet/plugins/chi/vsockets/vsockets.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/vsockets.go) supports both vsock and TCP fallbacks using `hvsock://` URLs.
- The **Cubelet** (agent) listens on vsock endpoints constructed in [`Cubelet/cmd/cubelet/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/cmd/cubelet/main.go) and registers ttrpc services via [`Cubelet/plugins/chi/vsockets/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/server.go).
- The **shim** connects using `HybridVSockDialer` from [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go) and creates ttrpc clients as shown in [`Cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/update.go).
- This architecture provides **kernel-mediated low latency**, **network independence**, and **isolated security** without requiring IP configuration inside the sandbox.

## Frequently Asked Questions

### What is the difference between vsock and regular TCP sockets in CubeSandbox?

**Vsock operates at the hypervisor level using the `AF_VSOCK` address family**, creating a direct communication channel between the host (CID 0) and the VM (CID 2) without traversing the network stack or requiring IP addresses. In contrast, TCP sockets require full network configuration, bridge interfaces, and packet encapsulation. The `Cubelet/plugins/chi/vsockets` package abstracts this difference by supporting `hvsock://` URLs that can resolve to either vsock or TCP connections for debugging purposes.

### Why does CubeSandbox use ttrpc instead of standard gRPC for shim communication?

**Ttrpc provides a significantly smaller footprint and lower overhead compared to standard gRPC**, as it eliminates the HTTP/2 transport layer and uses a lightweight framing protocol over protobuf. This is critical for shim-to-agent communication where minimal latency and binary efficiency are required. The implementation in [`Cubelet/services/cubebox/update.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/cubebox/update.go) uses `ttrpc.NewClient` and `ttrpc.NewServer` rather than gRPC equivalents to achieve these performance characteristics.

### How does the shim discover the correct vsock port to connect to the Cubelet agent?

**The Cubelet constructs the ttrpc address by appending `.ttrpc` to its configured gRPC address** during startup, as implemented in [`Cubelet/cmd/cubelet/main.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/cmd/cubelet/main.go). This generates a consistent addressing scheme (e.g., `vsock://2:1024.ttrpc`) that the shim can derive from the Cubelet's primary service endpoint. The shim then uses this address with `HybridVSockDialer` to establish the connection to the specific vsock port.

### Can the Hybrid VSock layer fall back to TCP for debugging purposes?

**Yes, the HybridVSock abstraction supports TCP fallback** for scenarios where vsock is unavailable or when debugging outside a hypervisor environment. The [`Cubelet/plugins/chi/vsockets/vsockets.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/vsockets.go) implementation parses `hvsock://` URLs and can resolve them to either real vsock connections (using `github.com/mdlayher/vsock`) or plain TCP connections based on configuration, allowing developers to test the shim-to-agent communication without requiring a full VM setup.