High-Concurrency API Design Patterns in the CubeAPI REST Gateway

CubeSandbox implements a four-layer architecture—combining Rust-based async services, high-performance reverse-proxying, and sub-domain isolation—to handle millions of concurrent REST requests with minimal latency and strict tenant isolation.

CubeSandbox’s CubeAPI REST Gateway is engineered to serve massive concurrent workloads while maintaining low latency and strong isolation between tenants. The architecture leverages Rust’s asynchronous runtime alongside intelligent routing and rate-limiting middleware. These high-concurrency API design patterns ensure that AI-agent workloads can scale horizontally without compromising performance or security.

The Four-Layer Concurrency Architecture

The gateway isolates concerns across four tightly integrated layers, each optimized for specific aspects of high-throughput HTTP handling.

Cube API Core (Rust/Axum)

The core HTTP service implements all sandbox, template, snapshot, and cluster operations using Rust with the Axum framework. All routes are mounted under the /cubeapi/v1 prefix, defined in CubeAPI/src/routes.rs.

A token-bucket rate-limiter middleware protects against abuse by inspecting the X-API-Key header (or falling back to IP/anonymous) and enforcing per-key quotas. If the quota is exceeded, the middleware returns a 429 Too Many Requests response immediately, preventing downstream saturation. The implementation resides in CubeAPI/src/middleware/rate_limit.rs.

Cube Proxy Multiplexer

The Cube Proxy is a tiny Rust HTTP server that multiplexes connections using tokio in a single-threaded, async configuration. It listens on 0.0.0.0:3000 (configurable via CUBE_API_URL) and directly forwards /:path requests to the Cube API socket via the host’s loopback interface.

This design avoids extra network hops by utilizing the host-gateway Docker bridge, minimizing latency between the public-facing edge and the application logic. The proxy startup logic is located in CubeAPI/src/main.rs.

Nginx Frontend

For production deployments, an nginx container provides TLS termination and static file serving for the Web UI. The server listens on port 12088 and proxies /cubeapi/ traffic to the host-gateway.

The configuration in deploy/one-click/webui/nginx.conf includes a location block that rewrites /cubeapi to /cubeapi/ and forwards all API requests to the Cube Proxy. This offloads TLS processing to a battle-tested server while keeping the API path within the high-performance Rust stack.

Gateway Sub-Domain Isolation

An optional gateway sub-domain isolation layer allows each sandbox to obtain its own origin (<port>-<sandboxId>.<domain>), ensuring WebSocket sessions never share a domain with other tenants.

The Web UI stores a gatewayDomain for each instance (see AgentSettingsDto in web/src/components/agents/AgentSettingsDialog.tsx). When configured, the UI constructs URLs like https://12088-<sandboxId>.cube.app/... as shown in web/src/api/client.ts (lines 46-52). This requires a wildcard DNS record and TLS certificate but enables strict CORS policies and per-sandbox rate limiting based on sub-domain API keys.

Request Flow and Rate Limiting

Understanding the request lifecycle reveals how these layers interact to manage concurrency:

1. Ingress: An incoming request hits nginx on port 12088, which reverse-proxies /cubeapi/ to the Cube Proxy on port 3000.
2. Forwarding: The Cube Proxy forwards the request to the Cube API process over the host’s loopback interface, avoiding NAT overhead.
3. Authentication: The Cube API extracts the X-API-Key header and checks the rate-limiter middleware.
4. Dispatch: If the token bucket permits the request, the Axum router dispatches it to the appropriate handler (e.g., sandbox creation, template listing).

This flow ensures that rate limiting occurs at the earliest possible stage, protecting the application layer from overload.

Scalability Mechanisms

The architecture addresses specific concurrency concerns through targeted optimizations:

• High Request Volume: The async Rust implementation (tokio) eliminates thread-per-connection overhead, using non-blocking I/O to handle millions of concurrent connections.
• Latency Reduction: Direct host-gateway routing between the proxy and API layers avoids network address translation (NAT) hops, keeping internal latency sub-millisecond.
• Tenant Isolation: Per-sandbox sub-domains combined with API-key-based rate limiting ensure that a single tenant cannot saturate the service or interfere with others.
• Operational Security: Nginx handles TLS termination and static asset delivery, isolating cryptographic overhead from the application servers.

Implementation Examples

The following cURL commands demonstrate the gateway’s dual routing capabilities:

# Basic health check (no API key required)

curl http://localhost:3000/cubeapi/v1/health

# Authenticated request with rate-limit key

curl -H "X-API-Key: my-key-123" \
     http://localhost:3000/cubeapi/v1/sandboxes

When utilizing gateway domain isolation, requests can target the sandbox-specific origin:

curl https://3000-<sandbox-id>.cube.app/cubeapi/v1/health

Both routes terminate at the same CubeAPI backend, but the sub-domain variant provides distinct origin isolation for WebSocket-dependent applications.

Summary

• Four-layer architecture: Cube API (Rust/Axum), Cube Proxy (tokio), Nginx frontend, and optional sub-domain isolation work together to handle high-concurrency workloads.
• Early rate limiting: Token-bucket middleware in CubeAPI/src/middleware/rate_limit.rs rejects excessive traffic before it reaches business logic.
• Zero-hop routing: The Cube Proxy communicates directly with the Cube API over the host loopback, minimizing latency.
• Tenant isolation: Optional sub-domain routing (<port>-<sandboxId>.<domain>) ensures complete traffic separation between sandboxes.

Frequently Asked Questions

How does CubeSandbox prevent a single tenant from overwhelming the API?

The gateway implements a token-bucket rate limiter in CubeAPI/src/middleware/rate_limit.rs that evaluates the X-API-Key header for every request. Each API key maintains an independent quota; when exceeded, the middleware returns HTTP 429 immediately, preventing downstream resource exhaustion.

Why is Rust with tokio used instead of a threaded server model?

The single-threaded async Rust implementation using tokio eliminates the memory overhead of thread-per-connection models. This allows the Cube Proxy to multiplex millions of concurrent connections efficiently, using non-blocking I/O operations that scale linearly with available CPU cores.

What is the purpose of the gateway sub-domain isolation feature?

Gateway sub-domain isolation assigns each sandbox a unique origin (<port>-<sandboxId>.<domain>), which prevents cross-origin resource sharing (CORS) complications and isolates WebSocket traffic. This pattern, configured in web/src/api/client.ts, ensures that browser-based clients treat each sandbox as a distinct security origin while enabling backend per-sandbox rate limiting.

How does the Nginx layer contribute to high-concurrency handling?

Nginx handles TLS termination and static file serving for the Web UI, offloading cryptographic operations from the Rust application servers. By proxying only dynamic /cubeapi/ traffic to the Cube Proxy, nginx allows the computationally intensive API layer to focus exclusively on request processing, improving overall throughput.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}