How Cube-Agent Forwards I/O Streams in TencentCloud CubeSandbox

Cube-agent forwards I/O streams by exposing a ttrpc service over a VM-side vsock channel, implementing RPC handlers that read from and write to container pseudo-terminals and pipes, and using an async copier to bridge low-level vsock traffic between the host shim and guest processes.

Cube-agent runs as PID 1 inside each MicroVM in the TencentCloud/CubeSandbox repository. Acting as the guest-side agent, it receives standard I/O (stdin, stdout, stderr) forwarded from the host-side containerd-shim-cube-rs over virtio-vsock, routing data to the appropriate container processes stored in the sandbox state.

Architecture Overview

The forwarding architecture relies on a vsock channel between the host shim and the guest agent. The host opens a connection to the guest’s vsock listener (VsockListener in agent/src/util.rs), which the agent wraps with a ttrpc server (AgentService in agent/src/rpc.rs).

The flow follows this path:

Host (shim) ──► vsock (ttrpc) ──► cube-agent (inside VM)
                                 │
          ┌──────────────────────┼───────────────────────┐
          │                      │                       │
          ▼                      ▼                       ▼
   WriteStream RPC      ReadStdout RPC          ReadStderr RPC
   (container stdin)    (container stdout)      (container stderr)

All stream objects reside inside the Process struct (defined in rustjail/src/process.rs) and are wrapped in Arc<Mutex<_>> to allow concurrent RPC calls to safely share the underlying file descriptors.

RPC-Based Stream Forwarding

The AgentService implementation in agent/src/rpc.rs exposes three primary RPC handlers for I/O operations. These methods map incoming ttrpc requests to the appropriate file descriptors or PTY masters belonging to the target container process.

Writing to Standard Input (do_write_stream)

When the shim sends data to a container’s stdin, it invokes the write_stdin RPC, handled by AgentService::do_write_stream (lines ≈ 616‑633 in agent/src/rpc.rs).

The handler performs the following steps:

1. Looks up the target process using sandbox.find_container_process.
2. Selects the appropriate writer:
   • If a PTY exists (term_master), it uses the PTY master file descriptor.
   • Otherwise, it falls back to the parent stdin pipe.
3. Writes the payload asynchronously: writer.lock().await.write_all(req.data.as_slice()).await.

Reading from Standard Output and Standard Error (do_read_stream)

For outbound data, the shim issues read_stdout or read_stderr RPCs, both handled by AgentService::do_read_stream (lines ≈ 644‑688 in agent/src/rpc.rs).

The logic mirrors the write path:

1. Locates the process via the sandbox.
2. Selects the reader:
   • PTY mode: Returns the PTY master stream.
   • Pipe mode: Returns the parent stdout or parent stderr pipe.
3. Invokes read_stream(reader, req.len as usize) to read exactly len bytes from the async reader.
4. Returns the bytes in a ReadStreamResponse to the host.

Terminal Resize Operations (tty_win_resize)

When containers run with a pseudo-terminal, the host can resize the terminal window via the tty_win_resize RPC (lines ≈ 1550‑1670 in agent/src/rpc.rs). This handler performs an ioctl on the PTY master file descriptor to adjust the window dimensions without interrupting the I/O stream.

Low-Level I/O Bridging with interruptable_io_copier

Beyond the ttrpc service, cube-agent handles raw vsock traffic for dedicated channels (such as metric exporting) using the interruptable_io_copier utility in agent/src/util.rs (lines ≈ 21‑57).

This async helper function performs a read-write loop between any AsyncRead source and AsyncWrite destination. It accepts a watch::Receiver<bool> shutdown signal, allowing the agent to abort the copy cleanly when the VM terminates.

From agent/src/main.rs (line ≈ 136), the agent initializes the copier at startup:

let _ = util::interruptable_io_copier(&mut reader, &mut writer, shutdown).await;

Here, reader and writer represent the vsock stream ends, creating a bridge between the low-level vsock pipe and the higher-level ttrpc service while respecting lifecycle signals.

End-to-End I/O Flow

The complete forwarding sequence operates as follows:

1. Sandbox Creation – The shim calls create_sandbox, and the agent initializes namespaces, mounts, and the Sandbox state.
2. Process Launch – During create_container, the agent constructs a Process (via rustjail), storing stdio pipes or the PTY master in the sandbox.
3. Stdin Write – The shim sends a WriteStreamRequest payload; do_write_stream writes bytes into the container’s stdin pipe or PTY master.
4. Stdout/Stderr Read – The shim issues ReadStreamRequest; do_read_stream pulls the requested byte count from the appropriate reader and returns a ReadStreamResponse.
5. Resizing – If using a PTY, the shim invokes tty_win_resize, which performs an ioctl on the PTY master to update terminal geometry.

This design ensures the host never requires direct filesystem or socket access to the guest; cube-agent acts as the sole authority for byte stream forwarding across the vsock boundary.

Summary

• Cube-agent exposes a ttrpc service over vsock to receive I/O commands from the host shim.
• do_write_stream (lines ≈ 616‑633 in agent/src/rpc.rs) writes incoming data to container stdin or PTY masters.
• do_read_stream (lines ≈ 644‑688 in agent/src/rpc.rs) reads requested byte counts from stdout/stderr pipes or PTY streams.
• interruptable_io_copier (lines ≈ 21‑57 in agent/src/util.rs) bridges raw vsock traffic with abortable async copy loops.
• Stream objects are stored in the Process struct (rustjail/src/process.rs) and protected by Arc<Mutex<_>> for thread-safe concurrent access.

Frequently Asked Questions

What transport protocol does cube-agent use for I/O forwarding?

Cube-agent uses ttrpc (a lightweight RPC framework) over virtio-vsock channels. The host-side shim opens a vsock connection to the guest agent, and all I/O operations are encapsulated in ttrpc requests such as write_stdin, read_stdout, and read_stderr.

How does cube-agent handle PTY I/O differently from pipe-based I/O?

When a container allocates a pseudo-terminal, cube-agent stores the term_master file descriptor in the Process struct. During I/O operations, do_write_stream and do_read_stream check for the presence of this PTY master first; if found, they use it as the reader or writer. If no PTY exists, they fall back to the standard parent_stdin, parent_stdout, or parent_stderr pipes.

What mechanism allows cube-agent to shut down I/O streams cleanly when the VM terminates?

The interruptable_io_copier function in agent/src/util.rs accepts a watch::Receiver<bool> shutdown signal. When the agent receives a termination signal, it triggers this watcher, causing the copier to abort its async read-write loop and close the vsock connection gracefully without dropping inflight data.

Where are the container I/O streams stored in cube-agent's memory?

Stream objects reside inside the Process struct defined in rustjail/src/process.rs. The agent stores parent_stdin, parent_stdout, parent_stderr, and optionally term_master as Arc<Mutex<>> wrapped types, allowing the AgentService RPC handlers to safely access these file descriptors concurrently from multiple async tasks.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}