How to Configure Resource Quotas and cgroup for Sandboxed Workloads in CubeSandbox

CubeSandbox enforces CPU and memory limits for Firecracker MicroVMs using Linux cgroups v2, configured through cubelet.yaml and applied at runtime by the agent's cgroup manager writing to cpu.max and memory.max controller files.

CubeSandbox isolates each sandboxed workload inside a lightweight Firecracker MicroVM and leverages the Linux kernel's cgroups v2 hierarchy for resource governance. To properly configure resource quotas and cgroup for sandboxed workloads, administrators must define node-level limits, ensure controller delegation, and understand how the runtime applies these constraints.

Architecture of Resource Quota Enforcement

CubeSandbox implements a three-layer resource control mechanism:

1. Configuration Layer – Node-wide limits are declared in configs/single-node/cubelet.yaml under the host.quota section, specifying maximum millicores and memory allocation.

2. Kernel Interface Layer – The host's cgroup v2 filesystem at /sys/fs/cgroup must expose the cpu, memory, and cpuset controllers, with proper delegation via cgroup.subtree_control.

3. Runtime Enforcement Layer – The Cubelet agent spawns cgroups for each sandbox through the rustjail cgroup manager, writing quota values directly to kernel controller files.

Defining Node-Level Resource Quotas

Resource quotas originate in the Cubelet configuration file. Edit configs/single-node/cubelet.yaml to set the upper bounds for CPU and memory consumption on the node.

host:
  quota:
    mcpu_limit: 64000        # 64 cores (expressed in millicores)

    mem_limit: "131072"      # 128 GiB (expressed in MiB)

    paused_resource_release_ratio: 0.5   # Release 50% of quota when paused

The mcpu_limit field accepts integer values representing millicores (1000 millicores = 1 CPU core), while mem_limit accepts strings or integers representing mebibytes (MiB). The optional paused_resource_release_ratio (0.0 to 1.0) determines what fraction of allocated resources are returned to the node pool when a sandbox enters a paused state.

Enabling cgroup v2 Controllers

Before CubeSandbox can create resource-constrained cgroups, the host kernel must expose the required controllers. The one-click installer (deploy/one-click/install.sh) performs pre-flight checks to verify that cpu, memory, and cpuset appear in /sys/fs/cgroup/cgroup.controllers.

On Ubuntu and Debian distributions, the cpu controller is frequently missing from the root cgroup. Enable it manually by writing to the subtree control file:

CGROOT="/sys/fs/cgroup"
if grep -q "^cpu$" "$CGROOT/cgroup.controllers"; then
  echo "+cpu" > "$CGROOT/cgroup.subtree_control"
else
  echo "CPU controller already enabled"
fi

Additionally, verify that /sys/fs/cgroup/cgroup.subtree_control contains cpu, memory, and cpuset entries to ensure child cgroups can inherit these controllers. Without this delegation, the agent cannot create sandbox-specific resource limits.

Runtime cgroup Creation and Enforcement

When a sandbox initializes, the Cubelet agent invokes the cgroup manager implemented in agent/rustjail/src/cgroups/fs/mod.rs. This module creates a new cgroup hierarchy under /sys/fs/cgroup and applies the configured quotas.

The agent uses the set_resource! macro to generate controller-specific write operations. For CPU limits, this expands to calls like set_cfs_quota, which writes to the cpu.max file:

set_resource!(cpu_controller, set_cfs_quota, cpu, quota);

This macro invocation writes the quota value to cpu.max using the cgroups v2 format <quota> 100000, where the quota represents microseconds of CPU time allocated per 100ms period. For example, 5000 millicores (5 cores) results in 500000 100000 written to the controller file.

Simultaneously, the memory limit is written to memory.max as a byte value. The kernel then enforces these limits, killing processes that exceed memory constraints or throttling CPU usage that surpasses the allocated quota.

Managing Quotas for Paused Sandboxes

CubeSandbox supports pausing sandboxes by shutting down their MicroVMs while preserving state. By default, paused sandboxes retain their full resource quota (paused_resource_release_ratio: 0.0), blocking those resources from other workloads.

To increase node density at the cost of slower resume times, adjust the release ratio in cubelet.yaml:

• 0.0 (default): Retain full quota allocation during pause
• 1.0: Release entire quota back to the node pool
• 0.5: Release half the allocated resources

When the sandbox resumes, the Cubelet reclaims the released resources and restores the original cgroup limits.

Verifying Quota Enforcement

After starting a sandbox, inspect the live cgroup filesystem to confirm limits are applied correctly. First, locate the cgroup path from the Cubelet runtime metadata, then examine the controller files:

CGRP=$(cat /run/cubelet/sandbox/<sandbox-id>/cgroup_path)
cat "$CGRP/cpu.max"      # Expected output: "50000 100000" for 500m CPU

cat "$CGRP/memory.max"   # Expected output: "536870912" for 512 MiB

The cpu.max file displays two values: the allowable time slice in microseconds and the period in microseconds (typically 100,000µs = 100ms). The memory.max file shows the absolute byte limit enforced by the kernel.

Monitor ongoing usage through the Cubelet node status API, which exposes QuotaCpuUsage and QuotaMemUsage fields defined in CubeAPI/src/services/sandboxes.rs to track actual consumption against configured limits.

Summary

• Resource quotas are defined in configs/single-node/cubelet.yaml using mcpu_limit (millicores) and mem_limit (MiB) fields
• cgroup v2 controllers must be enabled via /sys/fs/cgroup/cgroup.subtree_control before sandbox creation
• Runtime enforcement occurs in agent/rustjail/src/cgroups/fs/mod.rs through the set_resource! macro writing to cpu.max and memory.max
• Paused sandboxes can release resources based on paused_resource_release_ratio, trading density for resume latency
• Verification requires inspecting files under /sys/fs/cgroup/<sandbox-id>/ to confirm kernel-level enforcement

Frequently Asked Questions

Where are resource quotas configured in CubeSandbox?

Resource quotas are configured in the configs/single-node/cubelet.yaml file under the host.quota section. This YAML defines node-wide limits including mcpu_limit for CPU millicores and mem_limit for memory in MiB.

Why is the cpu controller missing on Ubuntu hosts?

Ubuntu and Debian images often ship with the cpu controller disabled in the root cgroup to maintain compatibility with legacy cgroup v1 tools. The CubeSandbox installer detects this condition and attempts to enable it by writing +cpu to /sys/fs/cgroup/cgroup.subtree_control, or you can enable it manually before installation.

How does CubeSandbox handle resource limits when a sandbox is paused?

When paused, a sandbox's MicroVM is shut down but its resource allocation behavior is controlled by the paused_resource_release_ratio setting. A value of 1.0 releases all quota back to the node, while 0.0 retains the full allocation. The default is 0.0, meaning paused workloads continue consuming their reserved CPU and memory capacity.

What file does the agent modify to set CPU quotas?

The agent modifies the cpu.max file within the sandbox's cgroup directory at /sys/fs/cgroup/<sandbox-id>/cpu.max. According to the implementation in agent/rustjail/src/cgroups/fs/mod.rs, the set_resource! macro writes the quota value in the format <quota_microseconds> 100000 to this file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}