# How to Implement Custom Device Management in CubeShim for Specialized Hardware Passthrough > Learn to implement custom device management in CubeShim for specialized hardware passthrough. Extend VFIO parsing, add device structs, and mount devices efficiently. - Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox) - Tags: how-to-guide - Published: 2026-07-03 --- **CubeShim supports specialized hardware passthrough by extending VFIO annotation parsing, adding device-specific structs to `SandboxConfig`, and modifying the sandbox builder to mount devices into the container namespace.** The CubeShim layer in the TencentCloud/CubeSandbox project provides a flexible framework for attaching host devices to sandboxed containers using Virtual Function I/O (VFIO) annotations. By extending the Rust structs in [`sandbox/config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sandbox/config.rs) and the parsing logic in [`sandbox/device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sandbox/device.rs), developers can implement custom device management for specialized hardware like FPGAs, GPUs, or proprietary PCI devices. ## Understanding CubeShim's VFIO Architecture CubeShim's device passthrough mechanism relies on a declarative annotation system that deserializes OCI runtime specifications into type-safe Rust structs. The architecture separates device definition from sandbox construction, allowing clean extensions for new hardware types. ### Core Configuration Components The device management system centers on three primary files in `CubeShim/shim/src/sandbox/`: - **[`config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.rs)** – Houses `SandboxConfig`, the central configuration object that aggregates all device annotations. It maintains vectors like `vfio_nets` and `vfio_disks`, plus lookup maps such as `vfio_disk_path_map` that correlate host device paths to internal indices. - **[`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs)** – Defines concrete device structs (`Device`, `DeviceDisk`) and implements helper methods including `driver_opt()` for mount options and `guest_pci_source()` for path resolution. - **[`sb.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sb.rs)** – Implements the sandbox builder that iterates over device lists stored in `SandboxConfig`, creates mount points (e.g., `/run/cube-containers/sandbox/blk-cube/`), and calls the lower-level hypervisor via [`cube_hypervisor.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/cube_hypervisor.rs) to bind VFIO devices into the guest. ### Device Injection Flow When a container launch request arrives at [`container/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/container/mod.rs), the shim executes the following sequence: 1. **Annotation parsing** – The shim scans OCI annotations for constants like `ANNO_VFIO_DISK` and `ANNO_VFIO_NET` defined in [`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs). 2. **Deserialization** – Using `serde`, JSON payloads convert into `Device` or `DeviceDisk` instances via `Utils::anno_to_obj`. 3. **Configuration aggregation** – `SandboxConfig` stores deserialized devices in typed vectors. 4. **Sandbox construction** – The builder walks configured device lists, applies driver-specific mount options, and registers devices with the hypervisor using `add_vfio_device()`. ## Extending CubeShim for Custom Hardware Types To implement custom device management for specialized hardware passthrough, you must extend the VFIO framework across four architectural layers. The following workflow demonstrates adding support for a PCI-based FPGA device. ### Step 1: Define Device Structs and Annotations First, declare a new annotation constant and struct in [`CubeShim/shim/src/sandbox/device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/sandbox/device.rs): ```rust // sandbox/device.rs pub const ANNO_VFIO_FPGA: &str = "cube.vfio.fpga"; #[derive(Clone, Debug, Default, Serialize, Deserialize)] pub struct DeviceFpga { #[serde(default)] pub id: String, #[serde(default)] pub bdf: String, #[serde(default)] pub sysfs_dev: String, #[serde(default)] pub profile: String, } ``` The `profile` field captures hardware-specific metadata (e.g., vendor type) that influences driver behavior. ### Step 2: Update Sandbox Configuration Extend `SandboxConfig` in [`CubeShim/shim/src/sandbox/config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/sandbox/config.rs) to store the new device type: ```rust // sandbox/config.rs #[derive(Default)] pub struct SandboxConfig { // Existing fields pub vfio_nets: Vec, pub vfio_disks: Vec, pub vfio_disk_path_map: HashMap, // New FPGA support pub vfio_fpgas: Vec, } ``` ### Step 3: Parse Custom Annotations Modify the configuration constructor to deserialize the new annotation: ```rust // sandbox/config.rs (inside impl SandboxConfig) let mut vfio_fpgas = Vec::new(); if let Some(fpga_anno) = conf.annotations.get(ANNO_VFIO_FPGA) { vfio_fpgas = Utils::anno_to_obj::>(fpga_anno)?; } SandboxConfig { // ... existing fields vfio_fpgas, } ``` ### Step 4: Implement Mount Point Logic Add helper methods to `DeviceFpga` for mount path generation and driver options: ```rust // sandbox/device.rs impl DeviceFpga { pub fn get_mount_point(&self) -> String { format!("{}{}", GUEST_PCI_MOUNT_DIR_PREFIX, self.bdf) } pub fn driver_opt(&self) -> Vec { vec![format!("fpga-profile={}", self.profile)] } } ``` ### Step 5: Integrate with Sandbox Builder Finally, update [`CubeShim/shim/src/sandbox/sb.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeShim/shim/src/sandbox/sb.rs) to process the custom devices during sandbox creation: ```rust // sandbox/sb.rs for fpga in self.conf.vfio_fpgas.iter() { // Create mount namespace entry let mount_point = fpga.get_mount_point(); self.mount(&mount_point, &fpga.sysfs_dev, fpga.driver_opt())?; // Register with hypervisor self.hypervisor.add_vfio_device(&fpga.sysfs_dev)?; } ``` ## Complete Implementation Example To use the extended FPGA support, specify the custom annotation in your container definition: ```json { "annotations": { "cube.vfio.fpga": "[{\"id\":\"fpga0\",\"bdf\":\"0000:03:00.0\",\"sysfs_dev\":\"/sys/bus/pci/devices/0000:03:00.0\",\"profile\":\"intel\"}]" } } ``` When launched, the shim deserializes the JSON array into `DeviceFpga` structs, creates the mount point `/run/cube-containers/sandbox/blk-cube/0000:03:00.0`, applies the `fpga-profile=intel` mount option, and binds the VFIO device into the guest namespace via [`container/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/container/mod.rs) orchestration. ## Summary - **CubeShim** uses a declarative VFIO annotation system defined in [`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs) and aggregated in [`config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.rs). - **Custom device management** requires extending `SandboxConfig` with new device vectors and updating the parsing logic in the configuration constructor. - **Mount point creation** and **driver options** are handled in [`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs) helper methods, while [`sb.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sb.rs) executes the actual hypervisor binding via `add_vfio_device()`. - **Type safety** is enforced through `serde` deserialization, ensuring runtime annotations map correctly to Rust structs. ## Frequently Asked Questions ### What is VFIO and why does CubeShim use it for hardware passthrough? VFIO (Virtual Function I/O) is a Linux kernel framework that exposes direct device access to user space while maintaining isolation between the host and guest. CubeShim leverages VFIO because it provides secure, high-performance device passthrough without requiring custom kernel modules, allowing containers to access specialized hardware like GPUs and FPGAs with near-native performance. ### How do I add support for multiple custom device types in CubeShim? Define distinct annotation constants (e.g., `ANNO_VFIO_GPU`, `ANNO_VFIO_NIC`) and corresponding structs in [`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs). Extend `SandboxConfig` in [`config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.rs) with separate vectors for each device type. Update the parsing logic to handle each annotation independently, then modify [`sb.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sb.rs) to iterate over each device category during sandbox construction. This approach maintains type safety and clear separation of concerns. ### Where should I handle driver-specific mount options for custom devices? Driver-specific mount options belong in the device struct's implementation within [`device.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/device.rs). Implement a `driver_opt()` method that returns a `Vec` of mount options based on device fields. The sandbox builder in [`sb.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/sb.rs) calls this method when invoking `self.mount()`, ensuring device-specific configuration stays encapsulated with the device definition rather than leaking into the builder logic. ### How does CubeShim ensure type safety when parsing device annotations? CubeShim uses `serde` derive macros (`Serialize`, `Deserialize`) on device structs combined with the `Utils::anno_to_obj` utility function. When parsing annotations in [`config.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/config.rs), the shim specifies the target type explicitly (e.g., `Utils::anno_to_obj::>`), causing deserialization failures at configuration time rather than runtime if the JSON structure doesn't match the expected schema.