# Strategies for High-Density Deployment of Thousands of CubeSandbox Instances

> Discover strategies for high-density CubeSandbox deployment using MicroVMs, Copy-on-Write, and kernel page sharing. Achieve low memory overhead for thousands of instances per host.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: performance
- Published: 2026-07-10

---

**High-density deployment of thousands of CubeSandbox instances relies on RustVMM-based MicroVMs, Copy-on-Write storage, kernel page sharing, and pre-allocated TAP device pools to achieve approximately 25 MiB of memory overhead per sandbox while supporting over 1,000 instances on a single host.**

CubeSandbox, developed by TencentCloud, is engineered to run thousands of isolated sandbox instances on a single bare-metal node. Achieving this level of density requires specific architectural choices and runtime configurations that minimize per-instance overhead. This guide explains the strategies for high-density deployment of thousands of sandbox instances based on the actual source code and benchmark data from the CubeSandbox repository.

## Architectural Foundations for Extreme Density

CubeSandbox achieves extreme density through a combination of virtualization technologies and resource-sharing mechanisms. Each component is designed to reduce the marginal cost of running an additional sandbox.

### MicroVM Isolation with RustVMM

Each sandbox runs in a lightweight **MicroVM** built on **RustVMM** and KVM. Unlike traditional virtual machines, these MicroVMs provide hardware-level isolation without the heavy overhead of a full operating system. This architecture is defined in [`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md) and enables the rapid instantiation required for high-density workloads.

### Copy-on-Write Storage

Sandbox filesystems utilize **XFS reflink** volumes with Copy-on-Write (CoW) semantics. As documented in the performance benchmark, this approach materializes only changed blocks, keeping the disk footprint minimal and enabling fast snapshot and clone operations. When deploying thousands of instances from a common template, the storage overhead remains nearly constant regardless of instance count.

### Kernel Page Sharing

The host kernel shares **identical pages** among all MicroVMs. According to the benchmark report in [`docs/blog/posts/2026-06-01-cubesandbox-perf-benchmark.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/blog/posts/2026-06-01-cubesandbox-perf-benchmark.md), this mechanism reduces the memory footprint of idle sandboxes to approximately **25 MiB** each. On a 375 GiB bare-metal node (Tencent Cloud BMI5), tests demonstrate a linear increase from ~21 MiB (100 sandboxes) to ~26 MiB (1000 sandboxes), allowing a single node to host **over 1,000 idle sandboxes** while leaving capacity for active workloads.

### Soft-Dirty Page Tracking

Snapshots record only pages dirtied since the last checkpoint using **soft-dirty page tracking**. While this feature is noted as landing in a future release, the benchmark report indicates it will reduce pause latency from ~550 ms to ~60 ms, significantly improving throughput during auto-pause operations.

### Pre-Allocated TAP Device Pools

The `network-agent` service maintains a pre-created pool of virtual NICs to eliminate per-sandbox TAP creation latency. The implementation in [`network-agent/internal/service/tap_lifecycle.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/internal/service/tap_lifecycle.go) includes an `InitPool` function that creates the target number of devices at startup:

```go
func (t *TapLifecycle) InitPool(target int) error {
    // Called at startup; creates `target` TAP devices.
    for i := 0; i < target; i++ {
        if err := t.createTap(); err != nil {
            return err
        }
    }
    return nil
}

```

### Stateless Control Plane

**CubeMaster**, **CubeAPI**, and **CubeProxy** hold no sandbox state; all coordination persists in Redis. This stateless design, documented in [`docs/architecture/overview.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/architecture/overview.md), allows horizontal scaling of the scheduler without adding per-sandbox state, making it possible to distribute thousands of instances across multiple nodes seamlessly.

## Runtime Configuration for Maximum Density

Specific configuration parameters in [`Cubelet/config/config.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/config/config.toml) and [`CubeMaster/config/cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/config/cubemaster.yaml) must be tuned to achieve optimal density.

### TAP Pool Sizing

Set `tap_init_num` in the Cubelet configuration to match or exceed your target sandbox count. For 1,000 sandboxes, configure:

```toml
[plugins."io.cubelet.internal.v1.network"]
tap_init_num = 1000

```

After modifying this value, restart the network-agent service to apply the change:

```bash
systemctl restart cube-sandbox-network-agent.service

```

### Resource Release Settings

Configure `host.quota.paused_resource_release_ratio` to `1.0` in [`cubemaster.yaml`](https://github.com/TencentCloud/CubeSandbox/blob/main/cubemaster.yaml) to release **all** CPU and memory resources when a sandbox is paused:

```yaml
host:
  quota:
    paused_resource_release_ratio: 1.0

```

This setting allows the scheduler to treat paused sandboxes as consuming zero resources, maximizing available capacity for new instances.

### Scheduler Tuning

Adjust the scheduler configuration to prevent hotspotting and optimize distribution:

```yaml
scheduler:
  priority_select_num: 3
  score:
    enable_scorers:
      - real_time_weighted_average
    resource_weights:
      mvm_num: 2
      local_create_num: 3
      quota_cpu_usage: 1
      quota_mem_usage: 1
    plugin_conf:
      real_time_weighted_average:
        weight: 1.0
        enable_weight_factors:
          - mvm_num
          - local_create_num
          - quota_cpu_usage
          - quota_mem_usage

```

Setting `priority_select_num` to `3` or higher allows the scheduler to select among the top-scored nodes rather than always picking the highest-scored node, preventing resource contention on individual hosts.

## Operational Strategies

Beyond architecture and configuration, specific operational practices maximize density and throughput.

### Auto-Pause and Auto-Resume

Idle sandboxes are automatically paused via the `POST /sandboxes/:id/pause` endpoint and resumed on demand. When combined with `paused_resource_release_ratio: 1.0`, this automatically frees resources for new instances, effectively increasing the total number of sandboxes that can be managed by a single host over time.

### Batch Creation and Concurrency Tuning

The `cube-bench` tool demonstrates that **20-concurrent** creation yields the optimal latency-throughput trade-off on a 96-core host. The benchmark data shows:

- **1 concurrent**: 55 ms per sandbox
- **10 concurrent**: 9.9 ms per sandbox
- **20 concurrent**: **5.5 ms** per sandbox (peak throughput)
- **50 concurrent**: 6.8 ms per sandbox (slightly higher due to queue depth)

Run the benchmark with:

```bash
cd examples/cube-bench
make
export E2B_API_URL=http://<control-node-ip>:3000
export E2B_API_KEY=e2b_000000
export CUBE_TEMPLATE_ID=<template-id>

./bin/cube-bench -c 20 -n 300 -w 3 -m create-only

```

### Snapshot-Based Workflows

Snapshot, clone, and rollback operations are near-instant (≈50 ms for snapshot, ≤10 ms per-instance when run concurrently). Using snapshots to branch workloads avoids the overhead of recreating sandboxes from templates. The benchmark scripts in [`examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py`](https://github.com/TencentCloud/CubeSandbox/blob/main/examples/snapshot-rollback-clone/bench_pause_resume_concurrency.py) demonstrate these operations at high concurrency.

### Multi-Node Scaling

When a single node reaches CPU or memory limits, add **compute-only nodes** that run only `Cubelet` and `network-agent`. The stateless control plane automatically balances scheduling across nodes via the configurable scheduler, as documented in [`docs/guide/multi-node-deploy.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/docs/guide/multi-node-deploy.md).

## Implementation Examples

### Python Bulk Creation with Auto-Pause

Use the Python SDK to create and pause sandboxes in batches:

```python
import os
from e2b_code_interpreter import Sandbox

os.environ["E2B_API_URL"] = "http://<control-node-ip>:3000"
os.environ["E2B_API_KEY"] = "e2b_000000"
os.environ["CUBE_TEMPLATE_ID"] = "<template-id>"

def create_many(n):
    sandboxes = []
    for _ in range(n):
        sb = Sandbox.create(template=os.environ["CUBE_TEMPLATE_ID"])
        sandboxes.append(sb)
    return sandboxes

# Create 1000 sandboxes

sandbox_list = create_many(1000)

# Pause all to free resources

for sb in sandbox_list:
    sb.pause()

```

### Configuring High-Density Parameters

Apply all configuration changes for a 1,000-sandbox deployment:

```bash

# 1. Configure Cubelet TAP pool

vi /usr/local/services/cubetoolbox/Cubelet/config/config.toml

# Add:

[plugins."io.cubelet.internal.v1.network"]
tap_init_num = 1000

# 2. Restart network agent

systemctl restart cube-sandbox-network-agent.service

# 3. Configure CubeMaster scheduler and resource release

vi /usr/local/services/cubetoolbox/CubeMaster/config/cubemaster.yaml

# Add:

scheduler:
  priority_select_num: 3
host:
  quota:
    paused_resource_release_ratio: 1.0

```

## Summary

- **MicroVM architecture** based on RustVMM provides hardware isolation with minimal overhead.
- **Copy-on-Write storage** and **kernel page sharing** keep per-instance memory overhead to approximately **25 MiB**.
- **Pre-allocate TAP devices** by setting `tap_init_num` to your target sandbox count before starting the network-agent.
- **Set `paused_resource_release_ratio` to 1.0** to free all resources from paused sandboxes, maximizing scheduling capacity.
- **Use 20-concurrent creation** for optimal throughput, yielding ~5.5 ms amortized latency per sandbox.
- **Leverage snapshots** for sub-100 ms branching operations instead of recreating instances from templates.
- **Scale horizontally** with compute-only nodes when single-host limits are reached; the stateless control plane handles distribution automatically.

## Frequently Asked Questions

### What is the memory overhead per CubeSandbox instance?

Each sandbox consumes approximately **25 MiB** of additional memory on the host after Copy-on-Write and kernel page sharing optimizations. Benchmarks on a 375 GiB bare-metal node show this overhead remains stable from 100 to 1,000 instances, enabling high-density deployment of thousands of sandbox instances on a single server.

### How do I prevent TAP device creation from becoming a bottleneck?

Configure `tap_init_num` in [`Cubelet/config/config.toml`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/config/config.toml) to pre-create a pool of virtual NICs equal to or greater than your maximum sandbox count. For example, set `tap_init_num = 1000` to support 1,000 sandboxes, then restart the network-agent service. This eliminates runtime TAP creation latency, which would otherwise impede high-density deployment.

### What concurrency level should I use for batch creation?

Use **20 concurrent** operations for optimal throughput. According to the `cube-bench` tool in `examples/cube-bench`, this concurrency level achieves the best amortized latency of approximately **5.5 ms** per sandbox on a 96-core host, compared to 55 ms at single concurrency or degraded performance at 50+ concurrency due to queue depth.

### How does auto-pause help achieve higher density?

When **auto-pause** is enabled and `paused_resource_release_ratio` is set to `1.0`, paused sandboxes release all CPU and memory resources to the scheduler. This allows the system to treat idle instances as consuming zero resources, effectively increasing the total number of sandboxes that can be managed across the cluster over time.