High-Density Deployment of Thousands of CubeSandbox Sandboxes on a Single Node: A Complete Guide

Deploying thousands of CubeSandbox sandboxes on a single node requires enabling Kernel Samepage Merging (mergeable=on), scaling the TAP device pool beyond the default 500 devices, and allocating fractional vCPU shares to minimize per-sandbox overhead.

CubeSandbox is engineered for high-concurrency workloads, supporting over 1,000 isolated sandboxes per physical node. Achieving this high-density deployment demands precise tuning of memory deduplication, network resource pools, and CPU allocation strategies to maintain VM-level isolation while keeping host resource consumption minimal.

Memory Optimization with KSM and Copy-on-Write

Memory efficiency is the primary constraint when scaling to thousands of sandboxes. CubeSandbox leverages Kernel Samepage Merging (KSM) and Copy-on-Write (CoW) to reduce the physical RAM footprint from gigabytes to megabytes per instance.

Enabling Mergeable Pages for KSM

In hypervisor/docs/memory.md, the mergeable=on option is explicitly designed for high-density scenarios. When enabled, the hypervisor marks guest memory as mergeable, allowing the host kernel to deduplicate identical pages across sandboxes.

ch-remote create --memory size=1G,mergeable=on,thp=off my-sandbox

According to the source documentation, this flag "can be used when trying to reach a higher density of VMs"【2†L53-L55】. Benchmark results demonstrate that with mergeable=on, a 1 GiB guest consumes only ≈ 25 MiB of real host RAM when idle, enabling approximately 1,000 sandboxes on a 375 GiB host【5†L244-L248】.

Copy-on-Write and Memory Over-Commit

CubeSandbox uses lazy allocation where guest RAM is not pre-allocated. Pages are only backed when written, allowing you to provision sandboxes with modest guest sizes (e.g., 2 GiB) while the actual host consumption remains low due to shared backing pages from a common template.

Set prefault=off (the default) to prevent eager memory allocation. As noted in hypervisor/docs/memory.md, enabling prefault "will consume … quickly" and should be avoided for high-density deployments【2†L66-L68】.

Transparent Huge Pages Considerations

While thp=on is the default, high-density deployments typically benefit from disabling it (thp=off) when memory is over-committed. Hugepages require fixed-size allocations that can increase memory pressure when hosting thousands of instances, so keep them disabled unless your specific workload demonstrably benefits from them.

Network and Compute Resource Management

Beyond memory, the underlying network interfaces and CPU scheduling require adjustment to support thousands of concurrent sandboxes.

Scaling the TAP Device Pool

By default, the Cubelet network agent pre-allocates 500 TAP devices (tap_init_num: 500). To support thousands of sandboxes, you must increase this value in network-agent/docs/CONFIGURATION.md and restart the network agent.

cat > /etc/cube/network-agent.yaml <<EOF
tap_init_num: 1200          # Must be >= target sandbox count

EOF
systemctl restart network-agent

The benchmark documentation confirms that failing to raise this limit prevents sandbox creation beyond the default pool size【5†L248-L251】.

Fractional vCPU Allocation

Each sandbox typically performs minimal CPU work, making fractional vCPU allocation essential. Use the --cpu option to request partial cores, allowing a 96-vCPU host to host over 2,000 sandboxes.

ch-remote create --cpu 0.1 --memory size=1G,mergeable=on my-sandbox

The "From Serverless to Agent" blog post notes that this configuration enables "over 1K sandboxes per node"【4†L99-L101】.

Template Reuse and NUMA Optimization

Efficient provisioning and hardware locality further stabilize high-density deployments.

Snapshot Reuse for Rapid Provisioning

Sandboxes should be created from a pre-built template (snapshot) to share read-only backing pages. When using --from-template or ch-remote create --from-snapshot, the host shares the same base image pages across all instances via CoW.

Keep a single read-only overlay for the template and provision new sandboxes via:

cubelet create-sandbox --from-template my-base-template --name sandbox-${i}

NUMA-Aware Memory Placement

On multi-socket hosts, pin sandbox memory to specific NUMA nodes to avoid cross-node traffic. The host_numa_node field in hypervisor/docs/memory.md controls memory-zone placement【2†L42-L49】.

ch-remote create --memory size=1G,mergeable=on,host_numa_node=0 my-sandbox

This ensures memory channels are exploited locally and reduces latency when the host has multiple memory domains.

Monitoring and Resource Limits

Operational visibility prevents resource exhaustion in dense environments.

Event Monitoring for Memory Pressure

Enable the event monitor to collect JSON metrics on sandbox lifecycle and memory pressure warnings from KSM.

ch-remote --event-monitor file=metrics.json create --name monitored-sandbox ...

As documented in README.md, the --event-monitor flag outputs structured events suitable for ingestion into Prometheus or Grafana【8†L198-L200】.

Resource Pool Sizing

Beyond TAP devices, check the resource_pool_size configuration in Cubelet for block device limits if hot-plugging many disks. Similar to the network pool, these defaults may require increase to match your sandbox count.

Complete High-Density Deployment Example

The following script demonstrates the complete workflow for launching 1,000 sandboxes with optimized settings:

# 1. Create a memory-efficient template with KSM enabled

cubelet create-template \
  --name high-density-template \
  --kernel /path/to/vmlinuz \
  --initramfs /path/to/initramfs.img \
  --rootfs /path/to/rootfs.ext4 \
  --cpu 0.1 \
  --memory size=1G,mergeable=on,thp=off

# 2. Scale the network agent TAP pool

cat > /etc/cube/network-agent.yaml <<EOF
tap_init_num: 1200
EOF
systemctl restart network-agent

# 3. Launch 1,000 sandboxes in parallel

for i in $(seq 1 1000); do
  cubelet create-sandbox \
    --from-template high-density-template \
    --name sandbox-${i} \
    --cpu 0.1 \
    --memory size=1G,mergeable=on \
    --net tap=auto &
done
wait

This configuration achieves the benchmarked density of approximately 1,000 sandboxes consuming only ~25 GiB total host memory through aggressive page sharing and lazy allocation【5†L244-L248】.

Summary

• Enable mergeable=on in hypervisor/docs/memory.md to activate KSM and reduce per-sandbox RAM to ~25 MiB.
• Increase tap_init_num in the network-agent configuration to support your target sandbox count beyond the default 500.
• Allocate fractional vCPUs (e.g., --cpu 0.1) to pack thousands of sandboxes onto available host cores.
• Disable prefault and hugepages to prevent eager memory consumption in over-committed scenarios.
• Use templates and snapshots to share read-only backing pages across all instances via CoW.
• Pin memory to NUMA nodes using host_numa_node to optimize locality on multi-socket hosts.
• Enable --event-monitor to track memory pressure and lifecycle events in real-time.

Frequently Asked Questions

How much physical RAM does each CubeSandbox actually consume?

With mergeable=on enabled, each idle sandbox with a 1 GiB guest size consumes approximately 25 MiB of real host RAM due to Kernel Samepage Merging and shared template pages. A 375 GiB host can accommodate roughly 1,000 sandboxes under these conditions【5†L244-L248】.

What is the default limit for TAP devices, and how do I increase it?

The default tap_init_num is 500 TAP devices. To support thousands of sandboxes, edit network-agent/docs/CONFIGURATION.md to set tap_init_num to match your target count (e.g., 1200), then restart the network-agent service【5†L248-L251】.

Should I enable hugepages for high-density deployments?

Generally no. While hugepages=on can improve boot performance, it increases memory pressure in dense deployments because each hugepage is a fixed-size allocation. Keep hugepages=off unless your specific workload demonstrates a clear benefit from hugepage usage【2†L21-L26】.

How do I prevent memory exhaustion when creating thousands of sandboxes?

Ensure prefault=off (the default) to prevent eager page allocation, and always use mergeable=on to enable KSM deduplication. Monitor JSON events via --event-monitor file=metrics.json to receive early warnings about memory pressure before the host triggers OOM【2†L66-L68】【8†L198-L200】.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}