How CubeSandbox Implements Resource Limits Using cgroups: A Deep Dive

CubeSandbox enforces resource limits by isolating each workload in a Linux cgroup using the Rust-based rustjail component and the cgroups-rs crate, which abstracts both cgroup v1 and v2 hierarchies to apply CPU, memory, PID, and I/O constraints.

CubeSandbox, TencentCloud's open-source sandbox runtime, leverages Linux cgroups (control groups) to provide hard resource boundaries for containerized workloads. The implementation resides primarily in the rustjail crate within the agent directory, where the cgroups-rs library bridges the gap between OCI container specifications and kernel-level resource controllers. This architecture ensures compatibility with both legacy cgroup v1 systems and modern unified cgroup v2 hierarchies.

Cgroup Creation and Hierarchy Detection

When a sandbox starts, CubeSandbox must first determine whether the host kernel uses the legacy cgroup v1 or the unified cgroup v2 mode before creating the appropriate control group structure.

Detecting Host Cgroup Version

The FsManager::new function in agent/rustjail/src/cgroups/fs/mod.rs auto-detects the hierarchy type by calling hierarchies::auto(), which returns a boxed trait object implementing the Hierarchy trait. This detection happens at lines 45-58, where the code checks h.v2() to determine the execution path.

Creating the Cgroup Object

Based on the detected version, the new_cgroup helper function instantiates the cgroup differently:

• For cgroup v2: Calls cgroups::Cgroup::new() with the hierarchy and path, allowing the library to translate kernel controller names into its unified model.
• For cgroup v1: Calls cgroups::Cgroup::new_with_specified_controllers(), explicitly passing CUBE_CONTROLLER (containing cpu, memory, pids, etc.) to create the legacy hierarchy.

// agent/rustjail/src/cgroups/fs/mod.rs
fn new_cgroup(h: Box<dyn cgroups::Hierarchy>, path: &str) -> Result<Cgroup> {
    let valid_path = path.trim_start_matches('/').to_string();
    if h.v2() {
        // unified cgroup v2
        return cgroups::Cgroup::new(h, &valid_path).context("create cgroup v2");
    }
    // legacy cgroup v1
    cgroups::Cgroup::new_with_specified_controllers(
        h,
        &valid_path,
        Some(CUBE_CONTROLLER.to_vec()),
    )
    .context("create cgroup v1")
}

Mapping OCI Resources to Kernel Controls

CubeSandbox translates the OCI LinuxResources specification into cgroups::Resources structs through specialized helper functions in mod.rs. Each resource type requires specific handling for v1 and v2 differences.

CPU Limits and Shares

The set_cpu_resources function configures CPU constraints by writing to the cpu controller. For cgroup v2, it converts traditional share values to the kernel's "weight" format using convert_shares_to_v2_value(). For both versions, it sets CFS quota, period, and realtime limits.

// agent/rustjail/src/cgroups/fs/mod.rs (lines 30-41)
if let Some(shares) = cpu.shares {
    let shares = if cg.v2() {
        convert_shares_to_v2_value(shares)
    } else { shares };
    if shares != 0 { cpu_controller.set_shares(shares)?; }
}
set_resource!(cpu_controller, set_cfs_quota, cpu, quota);
set_resource!(cpu_controller, set_cfs_period, cpu, period);

Memory Constraints and Swap Handling

The set_memory_resources function handles hard limits, soft limits, swap, kernel memory, and OOM-killer settings. It manages the order-dependent operations required by the kernel—specifically that memory limits must be set before swap limits when decreasing values. For cgroup v2, it converts swap values using convert_memory_swap_to_v2_value().

// agent/rustjail/src/cgroups/fs/mod.rs (lines 51-92)
if memory.limit.is_some() && swap != 0 {
    // Order-dependent: set swap first when decreasing
    mem_controller.set_memswap_limit(swap)?;
    set_resource!(mem_controller, set_limit, memory, limit);
} else {
    set_resource!(mem_controller, set_limit, memory, limit);
    // v2 conversion for swap value
    swap = if cg.v2() { convert_memory_swap_to_v2_value(swap, ...) } else { swap };
    if swap != 0 { mem_controller.set_memswap_limit(swap)?; }
}

PID Limits and Process Control

The set_pids_resources function uses the pids controller to enforce maximum process counts. It converts the OCI limit to either a specific value or MaxValue::Max when the limit is zero (unlimited).

// agent/rustjail/src/cgroups/fs/mod.rs (lines 20-27)
let v = if pids.limit > 0 {
    MaxValue::Value(pids.limit)
} else {
    MaxValue::Max
};
pid_controller.set_pid_max(v).context("failed to set pids resources")

Block I/O and Device Access

Additional helper functions—set_block_io_resources(), set_hugepages_resources(), and set_devices_resources()—populate the corresponding fields of the cgroups::Resources struct to enforce throttling limits on disk I/O, restrict hugepage usage, and control device access permissions.

Applying Resource Limits at Runtime

After the FsManager initializes the cgroup structure, the actual enforcement occurs in agent/rustjail/src/container.rs. The container struct holds an optional cgroup_manager field, and during container creation, it invokes set() to push the limits into the kernel.

if let Some(ref mut cgm) = self.cgroup_manager {
    // `r` is the OCI CreateContainerRequest carrying the LinuxResources
    cgm.set(&r, true)?;           // pushes all limits into the cgroup
}

The set method internally calls the resource-specific helpers described above, finally invoking cg.apply(&cg_res) to atomically apply all constraints to the kernel's cgroup filesystem.

Pre-flight Validation and Controller Setup

Before workloads start, the deploy/one-click/install.sh script performs pre-flight checks to ensure cgroup v2 controllers are available. It verifies that required controllers (cpu, memory, cpuset) exist in cgroup.controllers and attempts to enable the cpu controller in cgroup.subtree_control if missing.

# deploy/one-click/install.sh – cgroup v2 CPU preflight

if [[ -r "${cgroot}/cgroup.controllers" ]]; then
    controllers=$(cat "${cgroot}/cgroup.controllers")
    # fail if 'cpu' missing …

fi

# try to enable '+cpu' in subtree_control

if printf '+cpu\n' >"${cgroot}/cgroup.subtree_control"; then
    log "enabled '+cpu' on ${cgroot}/cgroup.subtree_control"
fi

This validation prevents runtime failures on Ubuntu/Debian cloud images where services like multipathd might block the cpu controller.

Summary

• CubeSandbox uses the cgroups-rs crate in agent/rustjail/src/cgroups/fs/mod.rs to abstract cgroup v1 and v2 hierarchies.
• The new_cgroup function detects the host version and creates appropriate controllers using either Cgroup::new() or Cgroup::new_with_specified_controllers().
• CPU, memory, PID, and I/O limits are translated from OCI specs to kernel-specific values, handling v2 conversions for shares and swap.
• Resource application occurs atomically via cgroup_manager.set() in agent/rustjail/src/container.rs.
• Installation scripts in deploy/one-click/install.sh verify that required v2 controllers are enabled in cgroup.subtree_control before deployment.

Frequently Asked Questions

How does CubeSandbox handle the difference between cgroup v1 and v2?

CubeSandbox detects the host hierarchy at runtime using hierarchies::auto(). For cgroup v2, it creates a unified cgroup using cgroups::Cgroup::new() and converts legacy share values to v2 "weight" values. For cgroup v1, it explicitly specifies required controllers (cpu, memory, pids) via new_with_specified_controllers() to create the legacy hierarchy. This dual-path approach ensures compatibility across different kernel configurations.

What happens if the CPU controller is not enabled in cgroup v2?

The pre-flight script in deploy/one-click/install.sh checks for the cpu controller in cgroup.controllers and attempts to enable it by writing +cpu to cgroup.subtree_control. If this fails, the sandbox initialization will fail with a clear error message, preventing workloads from starting without proper CPU isolation. This commonly occurs on cloud images where system services like multipathd hold the controller.

How are OCI container specifications translated to actual kernel limits?

The FsManager converts OCI LinuxResources into cgroups::Resources through specialized setters: set_cpu_resources() handles shares and CFS quotas, set_memory_resources() manages limits and swap with order-dependent logic, and set_pids_resources() configures process limits. Each function accounts for v1/v2 differences, such as converting CPU shares to v2 weight values or adjusting swap calculations for unified hierarchies.

Where does the actual resource application occur in the codebase?

The final application happens in agent/rustjail/src/container.rs where the container struct calls cgroup_manager.set(&r, true). This method invokes the resource-specific helpers in agent/rustjail/src/cgroups/fs/mod.rs, which write the constraints to the cgroup filesystem via the cgroups-rs crate's apply() method, atomically enforcing all limits in the kernel.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}