# How CubeSandbox Establishes Host-to-Guest Communication: Architecture and Implementation

> Discover how CubeSandbox achieves host-to-guest communication using AF_VSOCK sockets and tap devices for efficient, low-latency coordination between Cube Master and Cubelet.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: architecture
- Published: 2026-07-05

---

**CubeSandbox implements host-to-guest communication through a dual-channel architecture combining AF_VSOCK sockets for RPC control traffic and tap devices for TCP data plane connectivity, ensuring isolated yet low-latency coordination between the host Cube Master and guest Cubelet processes.**

CubeSandbox runs each sandbox as a lightweight Firecracker PVM (microVM), requiring robust communication pathways between the host management layer and guest workloads. As implemented in the TencentCloud/CubeSandbox repository, this host guest communication CubeSandbox architecture leverages two complementary channels: a VSOCK-based control plane for gRPC management operations and a tap-device-backed data plane for container TCP traffic. The design maintains strict VM isolation while enabling the host to manage snapshots, query state, and forward network connections to guest containers.

## Communication Architecture Overview

CubeSandbox partitions host-to-guest interaction into distinct control and data planes. This separation allows management RPCs to operate independently of application traffic, preventing resource contention during high-throughput operations.

The architecture consists of three primary components:

- **Control Plane**: Uses AF_VSOCK sockets (address family `vsock://3:1024`) to transport gRPC messages between the host and the guest Cubelet
- **Data Plane**: Employs virtual tap devices bridged to the guest NIC, with a host-side proxy binding specifically to the tap interface using `SO_BINDTODEVICE`
- **Hybrid Channel**: Provides fallback connectivity via `hvsock://` addresses that resolve to either VSOCK or Unix-domain sockets for environments where direct VSOCK is unavailable

## Control Plane: VSOCK RPC Implementation

The control plane enables RPC-style management operations including snapshot creation, template queries, and version information retrieval. This channel relies on the `vsockets` package built atop `github.com/mdlayher/vsock`.

### Guest-Side VSOCK Listener

Inside the guest, the Cubelet binary initializes the communication channel by starting a VSOCK listener on a fixed port (typically `1024`). The `ParseAndListen` function in [`Cubelet/plugins/chi/vsockets/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/server.go) handles address parsing and socket creation.

```go
// Inside the guest (Cubelet)
ln, err := vsockets.ParseAndListen("vsock://3:1024")
if err != nil { 
    log.Fatalf("listen error: %v", err)
}
defer ln.Close()
// Serve gRPC on ln...

```

The listener accepts connections from the host side, which then carry gRPC messages for Cubelet services.

### Host-Side VSOCK Client

The host process (CubeMaster or Cubelet) dials the guest using the `HybridVSockDialer` implemented in [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go). This dialer supports both standard VSOCK and hybrid address resolution.

```go
// Host side dialing
dialer := vsockets.HybridVSockDialer
conn, err := dialer.DialContext(
    ctx, 
    "vsock://3:1024", 
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil { 
    log.Fatalf("dial error: %v", err)
}
// Use conn for Cubelet gRPC client...

```

## Data Plane: TCP Proxy via Tap Devices

For ordinary TCP traffic exposed by containers inside the guest (such as web services), CubeSandbox utilizes a tap device approach managed by the **network-agent**.

### HostProxy Implementation

When a user requests port forwarding (e.g., `kubectl port-forward`), the host creates a `hostProxy` instance defined in [`network-agent/internal/service/hostproxy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/hostproxy.go). The proxy listens on a host-side IP and port, then forwards connections to the guest's IP address obtained from the network state (`firstGuestIP`).

The proxy creation follows this pattern:

```go
// HostProxy creation (simplified)
p, err := newHostProxy("0.0.0.0", 8080, "tap0", "169.254.68.6", 80, 30)
if err != nil { 
    log.Fatalf("proxy error: %v", err)
}
defer p.Close()

```

### Socket Binding with SO_BINDTODEVICE

The critical implementation detail ensuring traffic routes through the virtual NIC involves binding the host socket to the tap device using `SO_BINDTODEVICE`. This operation occurs in the dialer configuration within `newHostProxy`:

```go
dialer := &net.Dialer{
    Timeout: timeDurationSeconds(timeoutSeconds),
    Control: func(_, _ string, c syscall.RawConn) error {
        var ctrlErr error
        if err := c.Control(func(fd uintptr) {
            ctrlErr = unix.SetsockoptString(
                int(fd), 
                unix.SOL_SOCKET, 
                unix.SO_BINDTODEVICE, 
                tapName,
            )
        }); err != nil {
            return err
        }
        return ctrlErr
    },
}
backendConn, err := dialer.Dial("tcp", net.JoinHostPort(guestIP, strconv.Itoa(int(guestPort))))

```

The `network-agent` creates and manages the tap device lifecycle through [`network-agent/internal/service/tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/tap_lifecycle.go), while [`network-agent/internal/service/service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/service.go) coordinates proxy creation when port-mapping requests arrive.

## Hybrid Channel Support

For scenarios where the host cannot reach the guest via standard VSOCK—such as non-Firecracker environments or specific egress configurations—CubeSandbox provides a hybrid addressing scheme using `hvsock://` URLs. The `HybridVSockDialer` in [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go) resolves these addresses to either VSOCK or Unix-domain sockets, enabling the `cube-egress` sidecar functionality to communicate bidirectionally.

## Step-by-Step Communication Establishment

The complete lifecycle of host-to-guest communication follows this sequence:

1. **Guest initialization**: The Cubelet binary starts inside the Firecracker VM, invoking `ParseAndListen` to create the VSOCK listener on port `1024` and publishing network interface state via the network-agent.

2. **Control channel establishment**: The host queries `vsock://3:1024` using the hybrid client, establishing a gRPC connection for management RPCs.

3. **Port mapping request**: When exposing a guest service, the host calls the network-agent API, which returns the guest's IP address (extracted via [`network-agent/internal/service/local_service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/local_service.go)).

4. **Proxy instantiation**: The host creates a `hostProxy` bound to the tap device name (e.g., `tap0`), configuring the dialer with `SO_BINDTODEVICE` to ensure traffic traverses the virtual NIC.

5. **Bidirectional forwarding**: The proxy accepts host-side connections and dials the guest IP:port, performing `io.Copy` operations in both directions until the connections close.

6. **Fallback handling**: If VSOCK connectivity fails, the hybrid dialer automatically falls back to Unix-domain sockets for control operations.

## Summary

- **CubeSandbox** implements host-to-guest communication through a **dual-channel architecture** separating control and data plane traffic.
- The **control plane** uses **AF_VSOCK** sockets (`vsock://3:1024`) with gRPC, implemented in [`Cubelet/plugins/chi/vsockets/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/server.go) and [`client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/client.go).
- The **data plane** employs **tap devices** with host-side proxies that bind to the interface using **`SO_BINDTODEVICE`**, ensuring traffic routes through the virtual NIC.
- The **network-agent** coordinates tap device lifecycle ([`tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/tap_lifecycle.go)) and proxy creation ([`hostproxy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/hostproxy.go), [`service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/service.go)).
- **Hybrid VSOCK** (`hvsock://`) provides fallback connectivity for non-standard environments through the `HybridVSockDialer`.

## Frequently Asked Questions

### What transport protocol does CubeSandbox use for host-to-guest management RPCs?

CubeSandbox uses **AF_VSOCK** (virtual sockets) for management RPCs, specifically implementing a gRPC transport over VSOCK addresses formatted as `vsock://3:1024`. The guest listens on CID 3 (the standard Firecracker host CID) while the host dials this endpoint using the `HybridVSockDialer` from [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go).

### How does CubeSandbox ensure TCP traffic reaches the correct guest VM?

The system uses **tap device binding** with the `SO_BINDTODEVICE` socket option. When the `hostProxy` dials the guest IP address, it configures the `net.Dialer` with a control function that calls `unix.SetsockoptString(fd, SOL_SOCKET, SO_BINDTODEVICE, tapName)`. This forces the connection through the specific tap interface associated with that guest, preventing traffic leakage and ensuring isolation.

### What happens if VSOCK is not available in the runtime environment?

CubeSandbox falls back to **hybrid VSOCK** (`hvsock://`) addresses. The `HybridVSockDialer` in [`Cubelet/plugins/chi/vsockets/client.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/plugins/chi/vsockets/client.go) resolves these addresses to either standard VSOCK or Unix-domain sockets. This mechanism supports the `cube-egress` sidecar and other components when running in environments that do not support Firecracker's VSOCK implementation.

### Where is the guest IP address obtained when setting up port forwarding?

The host retrieves the guest IP from the **network-agent** state store, specifically through [`network-agent/internal/service/local_service.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/local_service.go). This component extracts the `firstGuestIP` from the guest's network configuration published during initialization, which the host then uses when creating the TCP proxy via `newHostProxy` in [`network-agent/internal/service/hostproxy.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/hostproxy.go).