Scalability Options for OmniRoute's Streaming Engine: Vertical Tuning and Horizontal Scaling

OmniRoute's streaming engine supports both vertical scaling through runtime environment variables that control timeouts, heartbeats, and concurrency limits, and horizontal scaling through a stateless architecture that allows multiple instances to run behind a load balancer sharing a central SQLite-backed quota store.

The streaming engine in diegosouzapw/OmniRoute handles Server-Sent Events (SSE) for chat completions, embeddings, and tool-heavy workflows. Its scalability options rely on a configuration-driven design where runtime constants govern everything from idle timeouts to back-pressure handling, enabling operators to fine-tune single-instance performance or scale out horizontally without code changes.

Core Configuration Levers

The engine exposes its scalability controls through environment variables defined in open-sse/config/constants.ts. Adjusting these changes behavior without requiring rebuilds or redeploys.

Stream Timeouts and Latency Control

Three primary constants govern how long the engine waits for upstream activity:

• STREAM_IDLE_TIMEOUT_MS – Controls the maximum silence period before an open SSE stream is forcibly closed. In open-sse/config/constants.ts:L14-L18, this defaults to 30 seconds to prevent "zombie" streams when a provider stalls after the first event.
• STREAM_READINESS_TIMEOUT_MS – Sets the time budget for the first non-ping SSE chunk (first-token latency). As defined in open-sse/config/constants.ts:L20-L23, defaults range from 5–30 seconds depending on the provider profile.
• STREAM_READINESS_MAX_TIMEOUT_MS – Caps adaptive extensions applied by the readiness resolver. According to open-sse/config/constants.ts:L25-L29, this defaults to 180 seconds, allowing extra padding for large histories or tool-heavy payloads while preventing infinite waits.

Background Job Concurrency

The BATCH_MAX_CONCURRENT environment variable controls how many batch-processor jobs may run simultaneously. Found in open-sse/services/batchProcessor.ts:L35-L38, this setting affects credential refresh, quota synchronization, and other background tasks. The default is 1, but increasing this allows higher throughput for authentication-heavy workloads.

Provider-Level Circuit Breakers

Each provider profile defines transient cooldowns, rate-limit cooldowns, and circuit-breaker thresholds that indirectly cap parallel requests. In open-sse/config/constants.ts:L60-L100, the PROVIDER_PROFILES object contains these definitions. Operators can override specific thresholds using OMNIROUTE_CIRCUIT_BREAKER_* variables to prevent cascading failures during provider outages.

Payload and Recovery Limits

• MAX_TOOLS_LIMIT – Set in open-sse/config/constants.ts:L30-L32, this defaults to 128 and prevents exploding payloads that could overwhelm the stream buffer.
• STREAM_RECOVERY – Controlled by STREAM_RECOVERY_ENABLED and related variables in open-sse/config/constants.ts:L62-L82, this optional hold-back mechanism (defaulting to HOLDBACK_MS=750 ms) reduces first-token latency for providers that truncate the opening SSE window, such as Claude Code.

Runtime Mechanisms

The engine implements these configurations through specific utility functions that manage the lifecycle of SSE connections.

Stall Watchdog and Abort Handling

The pipeWithDisconnect function in open-sse/utils/streamHandler.ts:L69-L74 and L99-L122 arms a stallTimeoutMs timer that resets on every raw upstream byte. If the timer fires, the engine emits a synthetic SSE error and aborts the upstream fetch. This timeout defaults to STREAM_IDLE_TIMEOUT_MS but can be overridden per request. The createStreamController function in open-sse/utils/streamHandler.ts:L10-L78 tracks client aborts and ensures pending-request counters are cleared exactly once, preventing client-side disconnects from triggering provider-level cooldowns that would reduce overall throughput.

Adaptive Readiness Policies

For "thinking-heavy" models like Codex GPT-5.x, the resolveStreamReadinessTimeout function in open-sse/utils/streamReadinessPolicy.ts:L71-L131 inspects request size—including history length, tool count, and payload characters—and optionally adds extra padding (e.g., +30000 ms). The final computed timeout is capped by STREAM_READINESS_MAX_TIMEOUT_MS, ensuring that large contexts get additional time without exceeding operator-defined limits.

SSE Heartbeat Management

To prevent strict proxies from closing connections during long "thinking" phases, the createSseHeartbeatTransform function in open-sse/utils/sseHeartbeat.ts:L64-L71 periodically enqueues keep-alive SSE chunks. The interval is controlled by SSE_HEARTBEAT_INTERVAL_MS (defaulting to 15,000 ms in open-sse/config/constants.ts:L36-L40). Setting this variable to 0 disables heartbeats entirely, reducing network churn for deployments using tolerant proxies.

Horizontal Scaling Architecture

Beyond vertical tuning, the streaming engine is architected for horizontal scaling across multiple instances.

Stateless Deployment Patterns

The engine maintains no per-connection state in memory that would block multiple processes. All coordination—including quota tracking, credential health, and back-off counters—is persisted in SQLite via the src/lib/db/ modules. Consequently:

• Deploy N identical OmniRoute containers behind a reverse proxy (Nginx, HAProxy, or cloud-native L7 routers).
• Each instance independently respects the same STREAM_IDLE_TIMEOUT_MS and back-off policies, so collective capacity scales linearly with N.
• Because quotas are stored centrally, the load balancer can safely route any client request to any replica without risking duplicate quota consumption.

Optional Redis Integration

For deployments requiring global rate-limit enforcement beyond SQLite's per-process view, the codebase includes an optional Redis-backed rate limiter at src/lib/rateLimiter/redis.ts. This code path is instrumented but disabled by default; enabling it allows cross-instance coordination of rate limits.

Configuration Examples

Tuning for High-Latency Reasoning Models

For deployments targeting models with extended reasoning phases, increase timeout ceilings and disable heartbeats:

# In your .env or deployment config

STREAM_IDLE_TIMEOUT_MS=60000
STREAM_READINESS_TIMEOUT_MS=30000
STREAM_READINESS_MAX_TIMEOUT_MS=240000
SSE_HEARTBEAT_INTERVAL_MS=0

Increasing Background Concurrency

To allow more simultaneous credential refreshes and quota syncs:

BATCH_MAX_CONCURRENT=5

Node.js SDK with Custom Timeouts

When using the client SDK, override per-request timeouts to match your deployment configuration:

import { createClient } from "omniroute/client";

const client = createClient({
  baseUrl: "https://my-omniroute-instance.com",
  streamIdleTimeoutMs: 45_000,
  streamReadinessTimeoutMs: 20_000,
});

const response = await client.chat.completions.create({
  model: "gpt-4o",
  stream: true,
  messages: [{ role: "user", content: "Explain quantum tunneling." }],
});

for await (const chunk of response) {
  console.log(chunk.choices[0].delta?.content ?? "");
}

Multi-Instance Docker Deployment

Scale horizontally using Docker Compose with four replicas:

version: "3.9"
services:
  omniroute:
    image: omniroute:latest
    env_file: .env
    deploy:
      mode: replicated
      replicas: 4
    ports:
      - "3000:3000"

Summary

• Vertical scaling is achieved by tuning environment variables in open-sse/config/constants.ts, including STREAM_IDLE_TIMEOUT_MS, BATCH_MAX_CONCURRENT, and STREAM_READINESS_MAX_TIMEOUT_MS.
• Horizontal scaling relies on the engine's stateless architecture; deploy multiple instances behind a load balancer, with SQLite handling quota persistence in src/lib/db/.
• Stall protection is implemented in open-sse/utils/streamHandler.ts via pipeWithDisconnect, which uses STREAM_IDLE_TIMEOUT_MS to detect and abort stalled upstream connections.
• Adaptive latency is calculated by resolveStreamReadinessTimeout in open-sse/utils/streamReadinessPolicy.ts, respecting STREAM_READINESS_MAX_TIMEOUT_MS for large payloads.
• Network stability is managed by createSseHeartbeatTransform in open-sse/utils/sseHeartbeat.ts, controllable via SSE_HEARTBEAT_INTERVAL_MS.

Frequently Asked Questions

How do I prevent proxy timeouts during long model reasoning phases?

Set SSE_HEARTBEAT_INTERVAL_MS to a value lower than your proxy's idle timeout (e.g., 10000 for 10 seconds), or set it to 0 if your proxy does not enforce idle timeouts. According to open-sse/config/constants.ts:L36-L40, the default is 15,000 ms. The createSseHeartbeatTransform function in open-sse/utils/sseHeartbeat.ts:L64-L71 emits synthetic keep-alive chunks to keep the connection alive.

Can I run multiple OmniRoute instances without sharing memory?

Yes. The streaming engine is stateless regarding connection handling; all quota and credential state is stored in SQLite (src/lib/db/). You can deploy N replicas behind a load balancer, and each instance will respect the same back-off policies defined in open-sse/config/constants.ts:L60-L100 without requiring shared memory or session affinity.

What happens if a provider stalls after sending the first token?

The pipeWithDisconnect function in open-sse/utils/streamHandler.ts:L99-L122 monitors every byte received from the upstream. If no data arrives within STREAM_IDLE_TIMEOUT_MS (default 30 seconds), the engine emits a synthetic SSE error and aborts the fetch, freeing the connection for new requests.

How do I increase throughput for high-volume chat applications?

Increase BATCH_MAX_CONCURRENT (found in open-sse/services/batchProcessor.ts:L35-L38) to allow more background credential refreshes and quota syncs to run in parallel. For extreme scale, enable the optional Redis rate limiter in src/lib/rateLimiter/redis.ts to coordinate limits across multiple instances, and deploy additional stateless replicas behind a load balancer.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}