# How to Configure Resource Quotas and cgroup for Sandboxed Workloads in CubeSandbox

> Learn how to configure resource quotas and cgroup for sandboxed workloads in CubeSandbox. Enforce CPU and memory limits for Firecracker MicroVMs using Linux cgroups v2.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-03

---

**CubeSandbox enforces CPU and memory limits for Firecracker MicroVMs using Linux cgroups v2, configured through [`cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet.yaml) and applied at runtime by the agent's cgroup manager writing to `cpu.max` and `memory.max` controller files.**

CubeSandbox isolates each sandboxed workload inside a lightweight Firecracker MicroVM and leverages the Linux kernel's cgroups v2 hierarchy for resource governance. To properly configure resource quotas and cgroup for sandboxed workloads, administrators must define node-level limits, ensure controller delegation, and understand how the runtime applies these constraints.

## Architecture of Resource Quota Enforcement

CubeSandbox implements a three-layer resource control mechanism:

1. **Configuration Layer** – Node-wide limits are declared in [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml) under the `host.quota` section, specifying maximum millicores and memory allocation.

2. **Kernel Interface Layer** – The host's cgroup v2 filesystem at `/sys/fs/cgroup` must expose the `cpu`, `memory`, and `cpuset` controllers, with proper delegation via `cgroup.subtree_control`.

3. **Runtime Enforcement Layer** – The Cubelet agent spawns cgroups for each sandbox through the rustjail cgroup manager, writing quota values directly to kernel controller files.

## Defining Node-Level Resource Quotas

Resource quotas originate in the Cubelet configuration file. Edit [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml) to set the upper bounds for CPU and memory consumption on the node.

```yaml
host:
  quota:
    mcpu_limit: 64000        # 64 cores (expressed in millicores)

    mem_limit: "131072"      # 128 GiB (expressed in MiB)

    paused_resource_release_ratio: 0.5   # Release 50% of quota when paused

```

The `mcpu_limit` field accepts integer values representing millicores (1000 millicores = 1 CPU core), while `mem_limit` accepts strings or integers representing mebibytes (MiB). The optional `paused_resource_release_ratio` (0.0 to 1.0) determines what fraction of allocated resources are returned to the node pool when a sandbox enters a paused state.

## Enabling cgroup v2 Controllers

Before CubeSandbox can create resource-constrained cgroups, the host kernel must expose the required controllers. The one-click installer ([`deploy/one-click/install.sh`](https://github.com/TencentCloud/CubeSandbox/blob/main/deploy/one-click/install.sh)) performs pre-flight checks to verify that `cpu`, `memory`, and `cpuset` appear in `/sys/fs/cgroup/cgroup.controllers`.

On Ubuntu and Debian distributions, the `cpu` controller is frequently missing from the root cgroup. Enable it manually by writing to the subtree control file:

```bash
CGROOT="/sys/fs/cgroup"
if grep -q "^cpu$" "$CGROOT/cgroup.controllers"; then
  echo "+cpu" > "$CGROOT/cgroup.subtree_control"
else
  echo "CPU controller already enabled"
fi

```

Additionally, verify that `/sys/fs/cgroup/cgroup.subtree_control` contains `cpu`, `memory`, and `cpuset` entries to ensure child cgroups can inherit these controllers. Without this delegation, the agent cannot create sandbox-specific resource limits.

## Runtime cgroup Creation and Enforcement

When a sandbox initializes, the Cubelet agent invokes the cgroup manager implemented in [`agent/rustjail/src/cgroups/fs/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/agent/rustjail/src/cgroups/fs/mod.rs). This module creates a new cgroup hierarchy under `/sys/fs/cgroup` and applies the configured quotas.

The agent uses the `set_resource!` macro to generate controller-specific write operations. For CPU limits, this expands to calls like `set_cfs_quota`, which writes to the `cpu.max` file:

```rust
set_resource!(cpu_controller, set_cfs_quota, cpu, quota);

```

This macro invocation writes the quota value to `cpu.max` using the cgroups v2 format `<quota> 100000`, where the quota represents microseconds of CPU time allocated per 100ms period. For example, 5000 millicores (5 cores) results in `500000 100000` written to the controller file.

Simultaneously, the memory limit is written to `memory.max` as a byte value. The kernel then enforces these limits, killing processes that exceed memory constraints or throttling CPU usage that surpasses the allocated quota.

## Managing Quotas for Paused Sandboxes

CubeSandbox supports pausing sandboxes by shutting down their MicroVMs while preserving state. By default, paused sandboxes retain their full resource quota (`paused_resource_release_ratio: 0.0`), blocking those resources from other workloads.

To increase node density at the cost of slower resume times, adjust the release ratio in [`cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubelet.yaml):

- **0.0** (default): Retain full quota allocation during pause
- **1.0**: Release entire quota back to the node pool
- **0.5**: Release half the allocated resources

When the sandbox resumes, the Cubelet reclaims the released resources and restores the original cgroup limits.

## Verifying Quota Enforcement

After starting a sandbox, inspect the live cgroup filesystem to confirm limits are applied correctly. First, locate the cgroup path from the Cubelet runtime metadata, then examine the controller files:

```bash
CGRP=$(cat /run/cubelet/sandbox/<sandbox-id>/cgroup_path)
cat "$CGRP/cpu.max"      # Expected output: "50000 100000" for 500m CPU

cat "$CGRP/memory.max"   # Expected output: "536870912" for 512 MiB

```

The `cpu.max` file displays two values: the allowable time slice in microseconds and the period in microseconds (typically 100,000µs = 100ms). The `memory.max` file shows the absolute byte limit enforced by the kernel.

Monitor ongoing usage through the Cubelet node status API, which exposes `QuotaCpuUsage` and `QuotaMemUsage` fields defined in [`CubeAPI/src/services/sandboxes.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeAPI/src/services/sandboxes.rs) to track actual consumption against configured limits.

## Summary

- **Resource quotas** are defined in [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml) using `mcpu_limit` (millicores) and `mem_limit` (MiB) fields
- **cgroup v2 controllers** must be enabled via `/sys/fs/cgroup/cgroup.subtree_control` before sandbox creation
- **Runtime enforcement** occurs in [`agent/rustjail/src/cgroups/fs/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/agent/rustjail/src/cgroups/fs/mod.rs) through the `set_resource!` macro writing to `cpu.max` and `memory.max`
- **Paused sandboxes** can release resources based on `paused_resource_release_ratio`, trading density for resume latency
- **Verification** requires inspecting files under `/sys/fs/cgroup/<sandbox-id>/` to confirm kernel-level enforcement

## Frequently Asked Questions

### Where are resource quotas configured in CubeSandbox?

Resource quotas are configured in the [`configs/single-node/cubelet.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/configs/single-node/cubelet.yaml) file under the `host.quota` section. This YAML defines node-wide limits including `mcpu_limit` for CPU millicores and `mem_limit` for memory in MiB.

### Why is the cpu controller missing on Ubuntu hosts?

Ubuntu and Debian images often ship with the `cpu` controller disabled in the root cgroup to maintain compatibility with legacy cgroup v1 tools. The CubeSandbox installer detects this condition and attempts to enable it by writing `+cpu` to `/sys/fs/cgroup/cgroup.subtree_control`, or you can enable it manually before installation.

### How does CubeSandbox handle resource limits when a sandbox is paused?

When paused, a sandbox's MicroVM is shut down but its resource allocation behavior is controlled by the `paused_resource_release_ratio` setting. A value of `1.0` releases all quota back to the node, while `0.0` retains the full allocation. The default is `0.0`, meaning paused workloads continue consuming their reserved CPU and memory capacity.

### What file does the agent modify to set CPU quotas?

The agent modifies the `cpu.max` file within the sandbox's cgroup directory at `/sys/fs/cgroup/<sandbox-id>/cpu.max`. According to the implementation in [`agent/rustjail/src/cgroups/fs/mod.rs`](https://github.com/TencentCloud/CubeSandbox/blob/main/agent/rustjail/src/cgroups/fs/mod.rs), the `set_resource!` macro writes the quota value in the format `<quota_microseconds> 100000` to this file.