# How to Achieve High-Density Deployment with CubeSandbox: Running Thousands of Sandboxes Per Node

> Learn to achieve high-density deployment with CubeSandbox, running thousands of sandboxes per node. Discover best practices for efficient resource utilization and minimal overhead.

- Repository: [Tencent Cloud/CubeSandbox](https://github.com/TencentCloud/CubeSandbox)
- Tags: best-practices
- Published: 2026-07-15

---

**You can run thousands of CubeSandbox instances per node by leveraging reflink-based storage pools, aggressive resource quotas, automated snapshot pruning, and kernel-level tuning to minimize overhead per sandbox.**

CubeSandbox is TencentCloud's container-like sandbox runtime designed for running isolated workloads at massive scale. When targeting high-density deployment scenarios with thousands of sandboxes per node, you must optimize storage, networking, and resource allocation to prevent node exhaustion. The following practices are derived directly from the TencentCloud/CubeSandbox source code.

## Optimize Storage with Reflink Pools

Storage efficiency determines how many sandboxes you can pack onto a single node. Use the **reflink-based pool** implemented in [`Cubelet/storage/pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool_withreflink.go) to create copy-on-write snapshots that share underlying blocks without duplicating data.

Enable reflink support on your host filesystem (XFS or Btrfs) and initialize the pool during Cubelet startup:

```go
// File: Cubelet/storage/pool_withreflink.go
pool, err := cubecow.NewPoolWithReflink("/var/lib/cubesandbox/pool")
if err != nil { 
    log.Fatal(err) 
}

```

This approach dramatically cuts the amount of data written per sandbox, allowing you to spin up thousands of instances from the same base image without exhausting disk space.

## Centralize Image Management

Network I/O becomes a bottleneck when thousands of sandboxes require the same template. Pull templates once per node using the **template-center** in [`CubeMaster/pkg/templatecenter/template_image.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/template_image.go), and rely on the image cache ([`cache.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/cache.go)) to avoid redundant downloads.

When a request arrives, the controller checks the cache before pulling:

```go
// File: CubeMaster/pkg/templatecenter/template_image.go
img, err := tmplCenter.GetOrPullImage(ctx, "nodejs-14")
if err != nil { 
    return err 
}

```

This reduces startup latency and prevents overwhelming your registry when scaling rapidly.

## Automate Snapshot Lifecycle Management

Unbounded snapshot growth consumes memory and inodes, eventually crashing the node. Configure the **snapshot reconciler** in [`CubeMaster/pkg/templatecenter/snapshot_reconciler.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_reconciler.go) to periodically prune expired snapshots.

Set a sensible Time-To-Live (TTL) in [`CubeMaster/pkg/templatecenter/snapshot_metrics.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_metrics.go) and run the cleanup loop every few minutes:

```go
// File: CubeMaster/pkg/templatecenter/snapshot_reconciler.go
go func() {
    ticker := time.NewTicker(10 * time.Minute)
    for range ticker.C {
        _ = reconciler.PruneExpiredSnapshots(ctx)
    }
}()

```

This prevents metadata bloat and ensures long-running nodes remain stable.

## Enforce Strict Resource Limits

Without limits, a single misbehaving sandbox can starve thousands of others. Configure conservative **cgroups** through the SDK by setting `cpuCount` and `memoryMB` in [`sdk/go/models.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sdk/go/models.go).

A typical high-density configuration allocates fractional resources:

```go
// File: sdk/go/sandbox.go
sb, err := client.NewSandbox(ctx, cubesandbox.SandboxOptions{
    TemplateID:  "nodejs-14",
    CPUCount:    0.1,           // 10% of a core
    MemoryMB:    64,            // 64 MiB
    IdleTimeout: 300 * time.Second,
})

```

These limits are enforced by the Cubelet runtime, guaranteeing node stability under heavy load.

## Configure Timeouts and Connection Pooling

Idle sandboxes waste resources that active workloads need. Set a short **idle timeout** (e.g., 300 seconds) via `sandbox.SetTimeout` in [`sdk/go/sandbox.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sdk/go/sandbox.go) to automatically terminate inactive instances.

For control plane efficiency, reuse HTTP connections across SDK operations. The Go SDK handles this in [`sdk/go/transport.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sdk/go/transport.go) through persistent keep-alive and HTTP/2 support:

```go
// File: sdk/go/transport.go
// The SDK reuses the HTTP client across operations to reduce 
// per-request overhead and TLS handshake latency when managing 
// thousands of sandbox endpoints.

```

This reduces connection churn when the control plane communicates with thousands of sandbox endpoints simultaneously.

## Tune the Host Environment and Filesystem

Kernel defaults are too low for thousands of concurrent sandboxes. Boot the node with **Linux kernel ≥ 5.10** and enable `CONFIG_FS_POSIX_ACL`. Tune system limits to prevent "too many files" errors:

- Increase `fs.inotify.max_user_watches`
- Raise `fs.file-max`
- Set `ulimit -n` to at least 200,000

Mount a dedicated **tmpfs** for each sandbox's writable layer using the logic in [`Cubelet/storage/hostdir.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/hostdir.go). This keeps I/O fast and avoids disk hotspots for short-lived processes:

```go
// File: Cubelet/storage/hostdir.go
// Mounts tmpfs for the sandbox's writable layer to provide 
// low-latency writes without backing disk contention.

```

## Implement Health Monitoring and Auto-Recovery

Individual sandboxes will crash under memory pressure or application errors. The **sandbox health monitor** in [`Cubelet/services/server/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/server/server.go) detects failed instances and automatically recreates them from the template pool.

This guarantees high availability without manual intervention during mass events like OOM kills or panic storms.

## Export Metrics for Capacity Planning

Visibility prevents saturation. Expose per-sandbox metrics via the Prometheus exporter in [`CubeMaster/pkg/templatecenter/snapshot_metrics.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/CubeMaster/pkg/templatecenter/snapshot_metrics.go):

```go
// File: CubeMaster/pkg/templatecenter/snapshot_metrics.go
prometheus.MustRegister(snapshotMetrics.ActiveSnapshots)
prometheus.MustRegister(snapshotMetrics.TotalBytes)

```

Aggregate these metrics at the node level to trigger horizontal scaling before resource exhaustion occurs.

## Summary

- **Use reflink pools** ([`pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/pool_withreflink.go)) on XFS/Btrfs to eliminate duplicate storage across sandboxes.
- **Cache templates** in the template-center ([`template_image.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/template_image.go)) to avoid redundant network pulls.
- **Prune snapshots automatically** via the reconciler ([`snapshot_reconciler.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/snapshot_reconciler.go)) to control metadata growth.
- **Set tight resource limits** (0.1 CPU, 64 MiB) through the SDK to maximize node utilization safely.
- **Configure short idle timeouts** ([`sandbox.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/sandbox.go)) and **reuse HTTP connections** ([`transport.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/transport.go)) to reduce overhead.
- **Tune kernel limits** and use **tmpfs** ([`hostdir.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/hostdir.go)) for writable layers to handle thousands of open files.
- **Enable health monitoring** ([`server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/server.go)) for automatic recovery from crashed sandboxes.

## Frequently Asked Questions

### What is the maximum number of sandboxes per node in CubeSandbox?

There is no hardcoded limit in the CubeSandbox source code, but practical density depends on available CPU, memory, and file descriptor limits. Production deployments regularly achieve **thousands of sandboxes per node** when using reflink storage, conservative resource quotas (0.1 CPU / 64 MiB per sandbox), and proper kernel tuning (`ulimit -n` ≥ 200,000).

### How does CubeSandbox minimize disk usage for thousands of sandboxes?

CubeSandbox uses **reflink-based copy-on-write** snapshots via [`Cubelet/storage/pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/storage/pool_withreflink.go). When you create a sandbox from a template, the filesystem creates a lightweight reference to existing blocks rather than copying the entire image. This allows thousands of sandboxes to share the same base data while writing only their unique changes, drastically reducing per-sandbox disk overhead.

### What happens when a CubeSandbox instance crashes?

The **sandbox health monitor** in [`Cubelet/services/server/server.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/Cubelet/services/server/server.go) continuously checks sandbox status. If a process exits unexpectedly or is killed by the OOM killer, the monitor automatically recreates the sandbox from the cached template pool. This ensures high availability without requiring manual intervention when individual sandboxes fail under high-density load.

### Which filesystem should I use for high-density CubeSandbox deployment?

Use **XFS** or **Btrfs** with reflink support enabled. The [`pool_withreflink.go`](https://github.com/TencentCloud/CubeSandbox/blob/main/pool_withreflink.go) implementation relies on filesystem-level copy-on-write capabilities to create efficient snapshots. Ensure your kernel is version 5.10 or later with `CONFIG_FS_POSIX_ACL` enabled, and mount the storage pool with appropriate mount options to enable reflink functionality.