# Scalability Options for OmniRoute's Streaming Engine: Vertical Tuning and Horizontal Scaling

> Explore OmniRoute's streaming engine scalability. Learn about vertical tuning, runtime variables, and horizontal scaling with stateless architecture for enhanced performance and reliability.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: performance
- Published: 2026-07-02

---

**OmniRoute's streaming engine supports both vertical scaling through runtime environment variables that control timeouts, heartbeats, and concurrency limits, and horizontal scaling through a stateless architecture that allows multiple instances to run behind a load balancer sharing a central SQLite-backed quota store.**

The streaming engine in `diegosouzapw/OmniRoute` handles Server-Sent Events (SSE) for chat completions, embeddings, and tool-heavy workflows. Its scalability options rely on a configuration-driven design where runtime constants govern everything from idle timeouts to back-pressure handling, enabling operators to fine-tune single-instance performance or scale out horizontally without code changes.

## Core Configuration Levers

The engine exposes its scalability controls through environment variables defined in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts). Adjusting these changes behavior without requiring rebuilds or redeploys.

### Stream Timeouts and Latency Control

Three primary constants govern how long the engine waits for upstream activity:

- **`STREAM_IDLE_TIMEOUT_MS`** – Controls the maximum silence period before an open SSE stream is forcibly closed. In `open-sse/config/constants.ts:L14-L18`, this defaults to 30 seconds to prevent "zombie" streams when a provider stalls after the first event.
- **`STREAM_READINESS_TIMEOUT_MS`** – Sets the time budget for the first non-ping SSE chunk (first-token latency). As defined in `open-sse/config/constants.ts:L20-L23`, defaults range from 5–30 seconds depending on the provider profile.
- **`STREAM_READINESS_MAX_TIMEOUT_MS`** – Caps adaptive extensions applied by the readiness resolver. According to `open-sse/config/constants.ts:L25-L29`, this defaults to 180 seconds, allowing extra padding for large histories or tool-heavy payloads while preventing infinite waits.

### Background Job Concurrency

The **`BATCH_MAX_CONCURRENT`** environment variable controls how many batch-processor jobs may run simultaneously. Found in `open-sse/services/batchProcessor.ts:L35-L38`, this setting affects credential refresh, quota synchronization, and other background tasks. The default is `1`, but increasing this allows higher throughput for authentication-heavy workloads.

### Provider-Level Circuit Breakers

Each provider profile defines transient cooldowns, rate-limit cooldowns, and circuit-breaker thresholds that indirectly cap parallel requests. In `open-sse/config/constants.ts:L60-L100`, the `PROVIDER_PROFILES` object contains these definitions. Operators can override specific thresholds using `OMNIROUTE_CIRCUIT_BREAKER_*` variables to prevent cascading failures during provider outages.

### Payload and Recovery Limits

- **`MAX_TOOLS_LIMIT`** – Set in `open-sse/config/constants.ts:L30-L32`, this defaults to `128` and prevents exploding payloads that could overwhelm the stream buffer.
- **`STREAM_RECOVERY`** – Controlled by `STREAM_RECOVERY_ENABLED` and related variables in `open-sse/config/constants.ts:L62-L82`, this optional hold-back mechanism (defaulting to `HOLDBACK_MS=750` ms) reduces first-token latency for providers that truncate the opening SSE window, such as Claude Code.

## Runtime Mechanisms

The engine implements these configurations through specific utility functions that manage the lifecycle of SSE connections.

### Stall Watchdog and Abort Handling

The `pipeWithDisconnect` function in `open-sse/utils/streamHandler.ts:L69-L74` and `L99-L122` arms a `stallTimeoutMs` timer that resets on every raw upstream byte. If the timer fires, the engine emits a synthetic SSE error and aborts the upstream fetch. This timeout defaults to `STREAM_IDLE_TIMEOUT_MS` but can be overridden per request. The `createStreamController` function in `open-sse/utils/streamHandler.ts:L10-L78` tracks client aborts and ensures pending-request counters are cleared exactly once, preventing client-side disconnects from triggering provider-level cooldowns that would reduce overall throughput.

### Adaptive Readiness Policies

For "thinking-heavy" models like Codex GPT-5.x, the `resolveStreamReadinessTimeout` function in `open-sse/utils/streamReadinessPolicy.ts:L71-L131` inspects request size—including history length, tool count, and payload characters—and optionally adds extra padding (e.g., `+30000` ms). The final computed timeout is capped by `STREAM_READINESS_MAX_TIMEOUT_MS`, ensuring that large contexts get additional time without exceeding operator-defined limits.

### SSE Heartbeat Management

To prevent strict proxies from closing connections during long "thinking" phases, the `createSseHeartbeatTransform` function in `open-sse/utils/sseHeartbeat.ts:L64-L71` periodically enqueues keep-alive SSE chunks. The interval is controlled by `SSE_HEARTBEAT_INTERVAL_MS` (defaulting to 15,000 ms in `open-sse/config/constants.ts:L36-L40`). Setting this variable to `0` disables heartbeats entirely, reducing network churn for deployments using tolerant proxies.

## Horizontal Scaling Architecture

Beyond vertical tuning, the streaming engine is architected for horizontal scaling across multiple instances.

### Stateless Deployment Patterns

The engine maintains **no per-connection state** in memory that would block multiple processes. All coordination—including quota tracking, credential health, and back-off counters—is persisted in SQLite via the `src/lib/db/` modules. Consequently:

- Deploy **N** identical OmniRoute containers behind a reverse proxy (Nginx, HAProxy, or cloud-native L7 routers).
- Each instance independently respects the same `STREAM_IDLE_TIMEOUT_MS` and back-off policies, so collective capacity scales linearly with **N**.
- Because quotas are stored centrally, the load balancer can safely route any client request to any replica without risking duplicate quota consumption.

### Optional Redis Integration

For deployments requiring global rate-limit enforcement beyond SQLite's per-process view, the codebase includes an optional Redis-backed rate limiter at [`src/lib/rateLimiter/redis.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/rateLimiter/redis.ts). This code path is instrumented but disabled by default; enabling it allows cross-instance coordination of rate limits.

## Configuration Examples

### Tuning for High-Latency Reasoning Models

For deployments targeting models with extended reasoning phases, increase timeout ceilings and disable heartbeats:

```bash

# In your .env or deployment config

STREAM_IDLE_TIMEOUT_MS=60000
STREAM_READINESS_TIMEOUT_MS=30000
STREAM_READINESS_MAX_TIMEOUT_MS=240000
SSE_HEARTBEAT_INTERVAL_MS=0

```

### Increasing Background Concurrency

To allow more simultaneous credential refreshes and quota syncs:

```bash
BATCH_MAX_CONCURRENT=5

```

### Node.js SDK with Custom Timeouts

When using the client SDK, override per-request timeouts to match your deployment configuration:

```javascript
import { createClient } from "omniroute/client";

const client = createClient({
  baseUrl: "https://my-omniroute-instance.com",
  streamIdleTimeoutMs: 45_000,
  streamReadinessTimeoutMs: 20_000,
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  stream: true,
  messages: [{ role: "user", content: "Explain quantum tunneling." }],
});

for await (const chunk of response) {
  console.log(chunk.choices[0].delta?.content ?? "");
}

```

### Multi-Instance Docker Deployment

Scale horizontally using Docker Compose with four replicas:

```yaml
version: "3.9"
services:
  omniroute:
    image: omniroute:latest
    env_file: .env
    deploy:
      mode: replicated
      replicas: 4
    ports:
      - "3000:3000"

```

## Summary

- **Vertical scaling** is achieved by tuning environment variables in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts), including `STREAM_IDLE_TIMEOUT_MS`, `BATCH_MAX_CONCURRENT`, and `STREAM_READINESS_MAX_TIMEOUT_MS`.
- **Horizontal scaling** relies on the engine's stateless architecture; deploy multiple instances behind a load balancer, with SQLite handling quota persistence in `src/lib/db/`.
- **Stall protection** is implemented in [`open-sse/utils/streamHandler.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/streamHandler.ts) via `pipeWithDisconnect`, which uses `STREAM_IDLE_TIMEOUT_MS` to detect and abort stalled upstream connections.
- **Adaptive latency** is calculated by `resolveStreamReadinessTimeout` in [`open-sse/utils/streamReadinessPolicy.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/streamReadinessPolicy.ts), respecting `STREAM_READINESS_MAX_TIMEOUT_MS` for large payloads.
- **Network stability** is managed by `createSseHeartbeatTransform` in [`open-sse/utils/sseHeartbeat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/sseHeartbeat.ts), controllable via `SSE_HEARTBEAT_INTERVAL_MS`.

## Frequently Asked Questions

### How do I prevent proxy timeouts during long model reasoning phases?

Set `SSE_HEARTBEAT_INTERVAL_MS` to a value lower than your proxy's idle timeout (e.g., `10000` for 10 seconds), or set it to `0` if your proxy does not enforce idle timeouts. According to `open-sse/config/constants.ts:L36-L40`, the default is 15,000 ms. The `createSseHeartbeatTransform` function in `open-sse/utils/sseHeartbeat.ts:L64-L71` emits synthetic keep-alive chunks to keep the connection alive.

### Can I run multiple OmniRoute instances without sharing memory?

Yes. The streaming engine is stateless regarding connection handling; all quota and credential state is stored in SQLite (`src/lib/db/`). You can deploy `N` replicas behind a load balancer, and each instance will respect the same back-off policies defined in `open-sse/config/constants.ts:L60-L100` without requiring shared memory or session affinity.

### What happens if a provider stalls after sending the first token?

The `pipeWithDisconnect` function in `open-sse/utils/streamHandler.ts:L99-L122` monitors every byte received from the upstream. If no data arrives within `STREAM_IDLE_TIMEOUT_MS` (default 30 seconds), the engine emits a synthetic SSE error and aborts the fetch, freeing the connection for new requests.

### How do I increase throughput for high-volume chat applications?

Increase `BATCH_MAX_CONCURRENT` (found in `open-sse/services/batchProcessor.ts:L35-L38`) to allow more background credential refreshes and quota syncs to run in parallel. For extreme scale, enable the optional Redis rate limiter in [`src/lib/rateLimiter/redis.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/rateLimiter/redis.ts) to coordinate limits across multiple instances, and deploy additional stateless replicas behind a load balancer.