How to Implement Custom Device Management in CubeShim for Specialized Hardware Passthrough

CubeShim supports specialized hardware passthrough by extending VFIO annotation parsing, adding device-specific structs to SandboxConfig, and modifying the sandbox builder to mount devices into the container namespace.

The CubeShim layer in the TencentCloud/CubeSandbox project provides a flexible framework for attaching host devices to sandboxed containers using Virtual Function I/O (VFIO) annotations. By extending the Rust structs in sandbox/config.rs and the parsing logic in sandbox/device.rs, developers can implement custom device management for specialized hardware like FPGAs, GPUs, or proprietary PCI devices.

Understanding CubeShim's VFIO Architecture

CubeShim's device passthrough mechanism relies on a declarative annotation system that deserializes OCI runtime specifications into type-safe Rust structs. The architecture separates device definition from sandbox construction, allowing clean extensions for new hardware types.

Core Configuration Components

The device management system centers on three primary files in CubeShim/shim/src/sandbox/:

• config.rs – Houses SandboxConfig, the central configuration object that aggregates all device annotations. It maintains vectors like vfio_nets and vfio_disks, plus lookup maps such as vfio_disk_path_map that correlate host device paths to internal indices.
• device.rs – Defines concrete device structs (Device, DeviceDisk) and implements helper methods including driver_opt() for mount options and guest_pci_source() for path resolution.
• sb.rs – Implements the sandbox builder that iterates over device lists stored in SandboxConfig, creates mount points (e.g., /run/cube-containers/sandbox/blk-cube/<guest-pci>), and calls the lower-level hypervisor via cube_hypervisor.rs to bind VFIO devices into the guest.

Device Injection Flow

When a container launch request arrives at container/mod.rs, the shim executes the following sequence:

1. Annotation parsing – The shim scans OCI annotations for constants like ANNO_VFIO_DISK and ANNO_VFIO_NET defined in device.rs.
2. Deserialization – Using serde, JSON payloads convert into Device or DeviceDisk instances via Utils::anno_to_obj.
3. Configuration aggregation – SandboxConfig stores deserialized devices in typed vectors.
4. Sandbox construction – The builder walks configured device lists, applies driver-specific mount options, and registers devices with the hypervisor using add_vfio_device().

Extending CubeShim for Custom Hardware Types

To implement custom device management for specialized hardware passthrough, you must extend the VFIO framework across four architectural layers. The following workflow demonstrates adding support for a PCI-based FPGA device.

Step 1: Define Device Structs and Annotations

First, declare a new annotation constant and struct in CubeShim/shim/src/sandbox/device.rs:

// sandbox/device.rs
pub const ANNO_VFIO_FPGA: &str = "cube.vfio.fpga";

#[derive(Clone, Debug, Default, Serialize, Deserialize)]
pub struct DeviceFpga {
    #[serde(default)]
    pub id: String,
    #[serde(default)]
    pub bdf: String,
    #[serde(default)]
    pub sysfs_dev: String,
    #[serde(default)]
    pub profile: String,
}

The profile field captures hardware-specific metadata (e.g., vendor type) that influences driver behavior.

Step 2: Update Sandbox Configuration

Extend SandboxConfig in CubeShim/shim/src/sandbox/config.rs to store the new device type:

// sandbox/config.rs
#[derive(Default)]
pub struct SandboxConfig {
    // Existing fields
    pub vfio_nets: Vec<Device>,
    pub vfio_disks: Vec<DeviceDisk>,
    pub vfio_disk_path_map: HashMap<String, u32>,
    
    // New FPGA support
    pub vfio_fpgas: Vec<DeviceFpga>,
}

Step 3: Parse Custom Annotations

Modify the configuration constructor to deserialize the new annotation:

// sandbox/config.rs (inside impl SandboxConfig)
let mut vfio_fpgas = Vec::new();
if let Some(fpga_anno) = conf.annotations.get(ANNO_VFIO_FPGA) {
    vfio_fpgas = Utils::anno_to_obj::<Vec<DeviceFpga>>(fpga_anno)?;
}

SandboxConfig {
    // ... existing fields
    vfio_fpgas,
}

Step 4: Implement Mount Point Logic

Add helper methods to DeviceFpga for mount path generation and driver options:

// sandbox/device.rs
impl DeviceFpga {
    pub fn get_mount_point(&self) -> String {
        format!("{}{}", GUEST_PCI_MOUNT_DIR_PREFIX, self.bdf)
    }
    
    pub fn driver_opt(&self) -> Vec<String> {
        vec![format!("fpga-profile={}", self.profile)]
    }
}

Step 5: Integrate with Sandbox Builder

Finally, update CubeShim/shim/src/sandbox/sb.rs to process the custom devices during sandbox creation:

// sandbox/sb.rs
for fpga in self.conf.vfio_fpgas.iter() {
    // Create mount namespace entry
    let mount_point = fpga.get_mount_point();
    self.mount(&mount_point, &fpga.sysfs_dev, fpga.driver_opt())?;
    
    // Register with hypervisor
    self.hypervisor.add_vfio_device(&fpga.sysfs_dev)?;
}

Complete Implementation Example

To use the extended FPGA support, specify the custom annotation in your container definition:

{
  "annotations": {
    "cube.vfio.fpga": "[{\"id\":\"fpga0\",\"bdf\":\"0000:03:00.0\",\"sysfs_dev\":\"/sys/bus/pci/devices/0000:03:00.0\",\"profile\":\"intel\"}]"
  }
}

When launched, the shim deserializes the JSON array into DeviceFpga structs, creates the mount point /run/cube-containers/sandbox/blk-cube/0000:03:00.0, applies the fpga-profile=intel mount option, and binds the VFIO device into the guest namespace via container/mod.rs orchestration.

Summary

• CubeShim uses a declarative VFIO annotation system defined in device.rs and aggregated in config.rs.
• Custom device management requires extending SandboxConfig with new device vectors and updating the parsing logic in the configuration constructor.
• Mount point creation and driver options are handled in device.rs helper methods, while sb.rs executes the actual hypervisor binding via add_vfio_device().
• Type safety is enforced through serde deserialization, ensuring runtime annotations map correctly to Rust structs.

Frequently Asked Questions

What is VFIO and why does CubeShim use it for hardware passthrough?

VFIO (Virtual Function I/O) is a Linux kernel framework that exposes direct device access to user space while maintaining isolation between the host and guest. CubeShim leverages VFIO because it provides secure, high-performance device passthrough without requiring custom kernel modules, allowing containers to access specialized hardware like GPUs and FPGAs with near-native performance.

How do I add support for multiple custom device types in CubeShim?

Define distinct annotation constants (e.g., ANNO_VFIO_GPU, ANNO_VFIO_NIC) and corresponding structs in device.rs. Extend SandboxConfig in config.rs with separate vectors for each device type. Update the parsing logic to handle each annotation independently, then modify sb.rs to iterate over each device category during sandbox construction. This approach maintains type safety and clear separation of concerns.

Where should I handle driver-specific mount options for custom devices?

Driver-specific mount options belong in the device struct's implementation within device.rs. Implement a driver_opt() method that returns a Vec<String> of mount options based on device fields. The sandbox builder in sb.rs calls this method when invoking self.mount(), ensuring device-specific configuration stays encapsulated with the device definition rather than leaking into the builder logic.

How does CubeShim ensure type safety when parsing device annotations?

CubeShim uses serde derive macros (Serialize, Deserialize) on device structs combined with the Utils::anno_to_obj utility function. When parsing annotations in config.rs, the shim specifies the target type explicitly (e.g., Utils::anno_to_obj::<Vec<DeviceFpga>>), causing deserialization failures at configuration time rather than runtime if the JSON structure doesn't match the expected schema.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}