How CubeSandbox Establishes Host-to-Guest Communication: Architecture and Implementation

CubeSandbox implements host-to-guest communication through a dual-channel architecture combining AF_VSOCK sockets for RPC control traffic and tap devices for TCP data plane connectivity, ensuring isolated yet low-latency coordination between the host Cube Master and guest Cubelet processes.

CubeSandbox runs each sandbox as a lightweight Firecracker PVM (microVM), requiring robust communication pathways between the host management layer and guest workloads. As implemented in the TencentCloud/CubeSandbox repository, this host guest communication CubeSandbox architecture leverages two complementary channels: a VSOCK-based control plane for gRPC management operations and a tap-device-backed data plane for container TCP traffic. The design maintains strict VM isolation while enabling the host to manage snapshots, query state, and forward network connections to guest containers.

Communication Architecture Overview

CubeSandbox partitions host-to-guest interaction into distinct control and data planes. This separation allows management RPCs to operate independently of application traffic, preventing resource contention during high-throughput operations.

The architecture consists of three primary components:

• Control Plane: Uses AF_VSOCK sockets (address family vsock://3:1024) to transport gRPC messages between the host and the guest Cubelet
• Data Plane: Employs virtual tap devices bridged to the guest NIC, with a host-side proxy binding specifically to the tap interface using SO_BINDTODEVICE
• Hybrid Channel: Provides fallback connectivity via hvsock:// addresses that resolve to either VSOCK or Unix-domain sockets for environments where direct VSOCK is unavailable

Control Plane: VSOCK RPC Implementation

The control plane enables RPC-style management operations including snapshot creation, template queries, and version information retrieval. This channel relies on the vsockets package built atop github.com/mdlayher/vsock.

Guest-Side VSOCK Listener

Inside the guest, the Cubelet binary initializes the communication channel by starting a VSOCK listener on a fixed port (typically 1024). The ParseAndListen function in Cubelet/plugins/chi/vsockets/server.go handles address parsing and socket creation.

// Inside the guest (Cubelet)
ln, err := vsockets.ParseAndListen("vsock://3:1024")
if err != nil { 
    log.Fatalf("listen error: %v", err)
}
defer ln.Close()
// Serve gRPC on ln...

The listener accepts connections from the host side, which then carry gRPC messages for Cubelet services.

Host-Side VSOCK Client

The host process (CubeMaster or Cubelet) dials the guest using the HybridVSockDialer implemented in Cubelet/plugins/chi/vsockets/client.go. This dialer supports both standard VSOCK and hybrid address resolution.

// Host side dialing
dialer := vsockets.HybridVSockDialer
conn, err := dialer.DialContext(
    ctx, 
    "vsock://3:1024", 
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil { 
    log.Fatalf("dial error: %v", err)
}
// Use conn for Cubelet gRPC client...

Data Plane: TCP Proxy via Tap Devices

For ordinary TCP traffic exposed by containers inside the guest (such as web services), CubeSandbox utilizes a tap device approach managed by the network-agent.

HostProxy Implementation

When a user requests port forwarding (e.g., kubectl port-forward), the host creates a hostProxy instance defined in network-agent/internal/service/hostproxy.go. The proxy listens on a host-side IP and port, then forwards connections to the guest's IP address obtained from the network state (firstGuestIP).

The proxy creation follows this pattern:

// HostProxy creation (simplified)
p, err := newHostProxy("0.0.0.0", 8080, "tap0", "169.254.68.6", 80, 30)
if err != nil { 
    log.Fatalf("proxy error: %v", err)
}
defer p.Close()

Socket Binding with SO_BINDTODEVICE

The critical implementation detail ensuring traffic routes through the virtual NIC involves binding the host socket to the tap device using SO_BINDTODEVICE. This operation occurs in the dialer configuration within newHostProxy:

dialer := &net.Dialer{
    Timeout: timeDurationSeconds(timeoutSeconds),
    Control: func(_, _ string, c syscall.RawConn) error {
        var ctrlErr error
        if err := c.Control(func(fd uintptr) {
            ctrlErr = unix.SetsockoptString(
                int(fd), 
                unix.SOL_SOCKET, 
                unix.SO_BINDTODEVICE, 
                tapName,
            )
        }); err != nil {
            return err
        }
        return ctrlErr
    },
}
backendConn, err := dialer.Dial("tcp", net.JoinHostPort(guestIP, strconv.Itoa(int(guestPort))))

The network-agent creates and manages the tap device lifecycle through network-agent/internal/service/tap_lifecycle.go, while network-agent/internal/service/service.go coordinates proxy creation when port-mapping requests arrive.

Hybrid Channel Support

For scenarios where the host cannot reach the guest via standard VSOCK—such as non-Firecracker environments or specific egress configurations—CubeSandbox provides a hybrid addressing scheme using hvsock:// URLs. The HybridVSockDialer in Cubelet/plugins/chi/vsockets/client.go resolves these addresses to either VSOCK or Unix-domain sockets, enabling the cube-egress sidecar functionality to communicate bidirectionally.

Step-by-Step Communication Establishment

The complete lifecycle of host-to-guest communication follows this sequence:

1. Guest initialization: The Cubelet binary starts inside the Firecracker VM, invoking ParseAndListen to create the VSOCK listener on port 1024 and publishing network interface state via the network-agent.

2. Control channel establishment: The host queries vsock://3:1024 using the hybrid client, establishing a gRPC connection for management RPCs.

3. Port mapping request: When exposing a guest service, the host calls the network-agent API, which returns the guest's IP address (extracted via network-agent/internal/service/local_service.go).

4. Proxy instantiation: The host creates a hostProxy bound to the tap device name (e.g., tap0), configuring the dialer with SO_BINDTODEVICE to ensure traffic traverses the virtual NIC.

5. Bidirectional forwarding: The proxy accepts host-side connections and dials the guest IP:port, performing io.Copy operations in both directions until the connections close.

6. Fallback handling: If VSOCK connectivity fails, the hybrid dialer automatically falls back to Unix-domain sockets for control operations.

Summary

• CubeSandbox implements host-to-guest communication through a dual-channel architecture separating control and data plane traffic.
• The control plane uses AF_VSOCK sockets (vsock://3:1024) with gRPC, implemented in Cubelet/plugins/chi/vsockets/server.go and client.go.
• The data plane employs tap devices with host-side proxies that bind to the interface using SO_BINDTODEVICE, ensuring traffic routes through the virtual NIC.
• The network-agent coordinates tap device lifecycle (tap_lifecycle.go) and proxy creation (hostproxy.go, service.go).
• Hybrid VSOCK (hvsock://) provides fallback connectivity for non-standard environments through the HybridVSockDialer.

Frequently Asked Questions

What transport protocol does CubeSandbox use for host-to-guest management RPCs?

CubeSandbox uses AF_VSOCK (virtual sockets) for management RPCs, specifically implementing a gRPC transport over VSOCK addresses formatted as vsock://3:1024. The guest listens on CID 3 (the standard Firecracker host CID) while the host dials this endpoint using the HybridVSockDialer from Cubelet/plugins/chi/vsockets/client.go.

How does CubeSandbox ensure TCP traffic reaches the correct guest VM?

The system uses tap device binding with the SO_BINDTODEVICE socket option. When the hostProxy dials the guest IP address, it configures the net.Dialer with a control function that calls unix.SetsockoptString(fd, SOL_SOCKET, SO_BINDTODEVICE, tapName). This forces the connection through the specific tap interface associated with that guest, preventing traffic leakage and ensuring isolation.

What happens if VSOCK is not available in the runtime environment?

CubeSandbox falls back to hybrid VSOCK (hvsock://) addresses. The HybridVSockDialer in Cubelet/plugins/chi/vsockets/client.go resolves these addresses to either standard VSOCK or Unix-domain sockets. This mechanism supports the cube-egress sidecar and other components when running in environments that do not support Firecracker's VSOCK implementation.

Where is the guest IP address obtained when setting up port forwarding?

The host retrieves the guest IP from the network-agent state store, specifically through network-agent/internal/service/local_service.go. This component extracts the firstGuestIP from the guest's network configuration published during initialization, which the host then uses when creating the TCP proxy via newHostProxy in network-agent/internal/service/hostproxy.go.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}