# High-Density Deployment of Thousands of CubeSandbox Sandboxes on a Single Node: A Complete Guide

> Learn how to deploy thousands of CubeSandbox sandboxes on a single node. Optimize kernel merging, scale TAP devices, and use fractional vCPUs for efficient high-density deployment. Get the complete guide.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: how-to-guide
- Published: 2026-07-03

---

**Deploying thousands of CubeSandbox sandboxes on a single node requires enabling Kernel Samepage Merging (`mergeable=on`), scaling the TAP device pool beyond the default 500 devices, and allocating fractional vCPU shares to minimize per-sandbox overhead.**

CubeSandbox is engineered for high-concurrency workloads, supporting over 1,000 isolated sandboxes per physical node. Achieving this **high-density deployment** demands precise tuning of memory deduplication, network resource pools, and CPU allocation strategies to maintain VM-level isolation while keeping host resource consumption minimal.

## Memory Optimization with KSM and Copy-on-Write

Memory efficiency is the primary constraint when scaling to thousands of sandboxes. CubeSandbox leverages Kernel Samepage Merging (KSM) and Copy-on-Write (CoW) to reduce the physical RAM footprint from gigabytes to megabytes per instance.

### Enabling Mergeable Pages for KSM

In [`hypervisor/docs/memory.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/docs/memory.md), the `mergeable=on` option is explicitly designed for high-density scenarios. When enabled, the hypervisor marks guest memory as mergeable, allowing the host kernel to deduplicate identical pages across sandboxes.

```bash
ch-remote create --memory size=1G,mergeable=on,thp=off my-sandbox

```

According to the source documentation, this flag "can be used when trying to reach a higher density of VMs"【2†L53-L55】. Benchmark results demonstrate that with `mergeable=on`, a 1 GiB guest consumes only **≈ 25 MiB** of real host RAM when idle, enabling approximately 1,000 sandboxes on a 375 GiB host【5†L244-L248】.

### Copy-on-Write and Memory Over-Commit

CubeSandbox uses lazy allocation where guest RAM is not pre-allocated. Pages are only backed when written, allowing you to provision sandboxes with modest guest sizes (e.g., 2 GiB) while the actual host consumption remains low due to shared backing pages from a common template.

Set `prefault=off` (the default) to prevent eager memory allocation. As noted in [`hypervisor/docs/memory.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/docs/memory.md), enabling `prefault` "will consume … quickly" and should be avoided for high-density deployments【2†L66-L68】.

### Transparent Huge Pages Considerations

While `thp=on` is the default, high-density deployments typically benefit from disabling it (`thp=off`) when memory is over-committed. Hugepages require fixed-size allocations that can increase memory pressure when hosting thousands of instances, so keep them disabled unless your specific workload demonstrably benefits from them.

## Network and Compute Resource Management

Beyond memory, the underlying network interfaces and CPU scheduling require adjustment to support thousands of concurrent sandboxes.

### Scaling the TAP Device Pool

By default, the Cubelet network agent pre-allocates **500 TAP devices** (`tap_init_num: 500`). To support thousands of sandboxes, you must increase this value in [`network-agent/docs/CONFIGURATION.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/docs/CONFIGURATION.md) and restart the network agent.

```bash
cat > /etc/cube/network-agent.yaml <<EOF
tap_init_num: 1200          # Must be >= target sandbox count

EOF
systemctl restart network-agent

```

The benchmark documentation confirms that failing to raise this limit prevents sandbox creation beyond the default pool size【5†L248-L251】.

### Fractional vCPU Allocation

Each sandbox typically performs minimal CPU work, making fractional vCPU allocation essential. Use the `--cpu` option to request partial cores, allowing a 96-vCPU host to host over 2,000 sandboxes.

```bash
ch-remote create --cpu 0.1 --memory size=1G,mergeable=on my-sandbox

```

The "From Serverless to Agent" blog post notes that this configuration enables "over 1K sandboxes per node"【4†L99-L101】.

## Template Reuse and NUMA Optimization

Efficient provisioning and hardware locality further stabilize high-density deployments.

### Snapshot Reuse for Rapid Provisioning

Sandboxes should be created from a **pre-built template** (snapshot) to share read-only backing pages. When using `--from-template` or `ch-remote create --from-snapshot`, the host shares the same base image pages across all instances via CoW.

Keep a single read-only overlay for the template and provision new sandboxes via:

```bash
cubelet create-sandbox --from-template my-base-template --name sandbox-${i}

```

### NUMA-Aware Memory Placement

On multi-socket hosts, pin sandbox memory to specific NUMA nodes to avoid cross-node traffic. The `host_numa_node` field in [`hypervisor/docs/memory.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/docs/memory.md) controls memory-zone placement【2†L42-L49】.

```bash
ch-remote create --memory size=1G,mergeable=on,host_numa_node=0 my-sandbox

```

This ensures memory channels are exploited locally and reduces latency when the host has multiple memory domains.

## Monitoring and Resource Limits

Operational visibility prevents resource exhaustion in dense environments.

### Event Monitoring for Memory Pressure

Enable the event monitor to collect JSON metrics on sandbox lifecycle and memory pressure warnings from KSM.

```bash
ch-remote --event-monitor file=metrics.json create --name monitored-sandbox ...

```

As documented in [`README.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/README.md), the `--event-monitor` flag outputs structured events suitable for ingestion into Prometheus or Grafana【8†L198-L200】.

### Resource Pool Sizing

Beyond TAP devices, check the `resource_pool_size` configuration in Cubelet for block device limits if hot-plugging many disks. Similar to the network pool, these defaults may require increase to match your sandbox count.

## Complete High-Density Deployment Example

The following script demonstrates the complete workflow for launching 1,000 sandboxes with optimized settings:

```bash

# 1. Create a memory-efficient template with KSM enabled

cubelet create-template \
  --name high-density-template \
  --kernel /path/to/vmlinuz \
  --initramfs /path/to/initramfs.img \
  --rootfs /path/to/rootfs.ext4 \
  --cpu 0.1 \
  --memory size=1G,mergeable=on,thp=off

# 2. Scale the network agent TAP pool

cat > /etc/cube/network-agent.yaml <<EOF
tap_init_num: 1200
EOF
systemctl restart network-agent

# 3. Launch 1,000 sandboxes in parallel

for i in $(seq 1 1000); do
  cubelet create-sandbox \
    --from-template high-density-template \
    --name sandbox-${i} \
    --cpu 0.1 \
    --memory size=1G,mergeable=on \
    --net tap=auto &
done
wait

```

This configuration achieves the benchmarked density of approximately **1,000 sandboxes consuming only ~25 GiB total host memory** through aggressive page sharing and lazy allocation【5†L244-L248】.

## Summary

- **Enable `mergeable=on`** in [`hypervisor/docs/memory.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/hypervisor/docs/memory.md) to activate KSM and reduce per-sandbox RAM to ~25 MiB.
- **Increase `tap_init_num`** in the network-agent configuration to support your target sandbox count beyond the default 500.
- **Allocate fractional vCPUs** (e.g., `--cpu 0.1`) to pack thousands of sandboxes onto available host cores.
- **Disable `prefault`** and **hugepages** to prevent eager memory consumption in over-committed scenarios.
- **Use templates and snapshots** to share read-only backing pages across all instances via CoW.
- **Pin memory to NUMA nodes** using `host_numa_node` to optimize locality on multi-socket hosts.
- **Enable `--event-monitor`** to track memory pressure and lifecycle events in real-time.

## Frequently Asked Questions

### How much physical RAM does each CubeSandbox actually consume?

With `mergeable=on` enabled, each idle sandbox with a 1 GiB guest size consumes approximately **25 MiB** of real host RAM due to Kernel Samepage Merging and shared template pages. A 375 GiB host can accommodate roughly 1,000 sandboxes under these conditions【5†L244-L248】.

### What is the default limit for TAP devices, and how do I increase it?

The default `tap_init_num` is **500** TAP devices. To support thousands of sandboxes, edit [`network-agent/docs/CONFIGURATION.md`](https://github.com/TencentCloud/CubeSandbox/blob/main/network-agent/docs/CONFIGURATION.md) to set `tap_init_num` to match your target count (e.g., 1200), then restart the network-agent service【5†L248-L251】.

### Should I enable hugepages for high-density deployments?

Generally **no**. While `hugepages=on` can improve boot performance, it increases memory pressure in dense deployments because each hugepage is a fixed-size allocation. Keep `hugepages=off` unless your specific workload demonstrates a clear benefit from hugepage usage【2†L21-L26】.

### How do I prevent memory exhaustion when creating thousands of sandboxes?

Ensure `prefault=off` (the default) to prevent eager page allocation, and always use `mergeable=on` to enable KSM deduplication. Monitor JSON events via `--event-monitor file=metrics.json` to receive early warnings about memory pressure before the host triggers OOM【2†L66-L68】【8†L198-L200】.