# Limitations of OmniRoute's Current Streaming Implementation: SSE Pipeline Constraints Explained

> Explore OmniRoute's SSE pipeline limitations, including rigid timeouts, inconsistent handling, and data integrity risks when proxying LLM streams. Learn how constraints impact performance.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: internals
- Published: 2026-07-02

---

**OmniRoute's Server-Sent Events (SSE) pipeline contains several hard-coded design choices that create rigid timeout behaviors, format-specific handling inconsistencies, and data integrity risks when proxying LLM streams.**

OmniRoute is an open-source LLM routing gateway that streams responses from upstream providers to clients using a custom SSE transform pipeline. While the implementation in [`open-sse/utils/stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/stream.ts) supports multiple provider formats, its current architecture imposes specific constraints that can cause unexpected terminations, silent data loss, and client compatibility issues. Understanding these boundaries is essential for debugging premature disconnects and configuring reliable production deployments.

## Hard-Coded Timeout Constraints

The streaming layer enforces two distinct timeout windows that are not automatically negotiated with upstream providers.

### Idle Timeout Termination

The pipeline watches for data arrival using `STREAM_IDLE_TIMEOUT_MS` (default approximately 30 seconds) configured in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts). If no bytes arrive within this window, the `createSSEStream` function (lines 591‑620 in [`open-sse/utils/stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/stream.ts)) forcibly closes the connection and emits a `StreamIdleTimeoutError`.

This watchdog runs in the `start` hook of the transform stream. For providers that enter long reasoning phases before emitting the first token—common with large prompt completions or tool-heavy requests—the client receives a gateway-timeout error even though the upstream remains active.

### Readiness Timeout Handling

The same idle timer enforces a *first-event* window via `STREAM_READINESS_TIMEOUT_MS` (defined at lines 14‑23 in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts)). Slow-to-first-token models can hit this readiness timeout before generating any output, causing the stream to abort before the first SSE event is dispatched.

Unit tests in [`tests/unit/transform-stream-hwm.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/transform-stream-hwm.test.ts) (lines 139‑148) explicitly verify that this timeout fires when the stream remains empty, confirming the behavior is intentional rather than ephemeral.

## Format-Specific Handling Issues

OmniRoute normalizes provider differences through translation logic that occasionally violates client expectations.

### Synthetic Claude Empty Response Injection

When the Anthropic provider returns no content blocks, the `emitSyntheticClaudeEmptyResponse` function (lines 813‑847 in [`open-sse/utils/stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/stream.ts)) fabricates a placeholder response. This injection adds latency, generates warning logs, and masks genuine upstream failures, complicating root-cause analysis for empty reply scenarios.

### Strict `[DONE]` Terminator Handling

The pipeline deliberately strips the OpenAI-style `[DONE]` terminator for non-OpenAI target formats (lines 444‑447). Clients relying on this explicit signal—including custom SDKs that wait for the sentinel line—will hang indefinitely because the termination event never arrives. The `shouldEmitDoneTerminator` logic assumes all non-OpenAI consumers parse provider-specific end signals like `response.completed`.

### Keep-Alive Event Stripping

SSE keep-alive frames (`event: keepalive`) are removed during transformation (lines 511‑557). In strict networking environments or corporate proxies that require periodic traffic to maintain NAT bindings, this stripping can trigger premature TCP disconnects because the connection appears idle despite being healthy.

## Data Integrity and Parsing Limitations

Several hard-coded parsers risk silently altering or truncating payload content.

### Tool-Call Parsing Quirks

Textual tool-call detection operates only within the `content` field of `delta` objects via `applyTextualToolCallStreamingGuard` (lines 554‑580). Multi-line tool-call payloads or complex nested function arguments may be mis-detected, resulting in malformed or dropped `function_call` chunks during streaming.

### Duplicate Sequence Number Filtering

For the Responses API, a hard-coded watermark (`lastSeenResponsesSequenceNumber`) discards any event where `sequence_number` is less than or equal to the last seen value (lines 691‑698). While this prevents replay attacks, it can also suppress legitimate retries where a provider legitimately repeats a sequence number, causing those events to vanish without client notification.

### Buffer Truncation Risks

The stream buffers up to `STREAM_SUMMARY_TEXT_LIMIT` (64 KB) before truncating excess data (lines 665‑672). Lengthy code completions or document generation tasks that exceed this limit are silently clipped, producing incomplete JSON or truncated text without raising an explicit error.

## Provider Compatibility Constraints

Translation mode (`STREAM_MODE.TRANSLATE`) supports only a subset of provider formats. Detection logic in `isResponsesSSE` and `isClaudeSSE` (lines 1260‑1285) filters non-standard event types, meaning providers emitting custom metadata or proprietary SSE extensions have those frames dropped or transformed. Adding a new provider often requires patching [`stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/stream.ts) directly, violating the "drop-in" extensibility model.

## Workarounds and Mitigation Strategies

While the limitations are architecturally embedded, environment variables and runtime options provide partial relief.

**Increase Timeout Values**: Set `STREAM_IDLE_TIMEOUT_MS` and `STREAM_READINESS_TIMEOUT_MS` via environment variables before server initialization to accommodate slow providers.

**Disable Synthetic Injection**: Pass `suppressThinkClose: true` in the `StreamOptions` object when invoking `createSSEStream` to prevent artificial empty-response generation for Claude streams.

**Implement Client-Side Termination**: For non-OpenAI formats, listen for provider-specific end events (e.g., `message_stop`) rather than waiting for `[DONE]`, or handle stream closure via the reader's `close()` signal.

```typescript
// Example: Creating a stream with extended idle timeout
import { createSSEStream } from '@/open-sse/utils/stream.ts';

const sse = createSSEStream({
  mode: 'translate',
  sourceFormat: 'openai',
  targetFormat: 'anthropic',
  suppressThinkClose: true,
  onFailure: (payload) => {
    console.error('Stream failed:', payload);
  },
});

// Detect idle-timeout errors explicitly
sse.readable
  .pipeThrough(new TextDecoderStream())
  .pipeTo(new WritableStream({
    write(chunk) {
      console.log('Chunk →', chunk);
    },
    close() {
      console.log('Stream closed gracefully');
    },
    abort(err) {
      if (err?.name === 'StreamIdleTimeoutError') {
        console.warn('Provider stalled – consider retrying with longer STREAM_IDLE_TIMEOUT_MS');
      }
    },
  }));

```

```typescript
// Example: Handling missing [DONE] terminators for custom clients
fetch('/api/v1/chat/completions', { 
  method: 'POST', 
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ model: 'claude-3-opus', messages: [] })
})
  .then(res => {
    const reader = res.body?.getReader();
    const decoder = new TextDecoder();
    
    // Read until connection closes instead of waiting for [DONE]
    function read() {
      reader?.read().then(({ done, value }) => {
        if (done) {
          console.log('Stream ended (no [DONE] terminator expected for this format)');
          return;
        }
        console.log(decoder.decode(value, { stream: true }));
        read();
      });
    }
    read();
  });

```

## Summary

OmniRoute's SSE implementation in [`open-sse/utils/stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/stream.ts) imposes specific architectural constraints that operators must navigate:

- **Rigid timeouts**: `STREAM_IDLE_TIMEOUT_MS` and `STREAM_READINESS_TIMEOUT_MS` can abort legitimate long-running requests after ~30 seconds of inactivity or initial delay.
- **Format normalization side effects**: The pipeline drops `[DONE]` terminators for non-OpenAI targets, strips keep-alive events, and injects synthetic Claude responses that mask errors.
- **Data integrity risks**: The 64 KB high-water mark truncates large payloads, sequence-number deduplication can lose valid retries, and textual tool-call parsing fails on complex multi-line arguments.
- **Extension friction**: Hard-coded format detection in the translation layer requires source patches to support new providers emitting non-standard SSE events.

These constraints are verified in [`tests/unit/transform-stream-hwm.test.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/tests/unit/transform-stream-hwm.test.ts) and configurable primarily through environment variables in [`open-sse/config/constants.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/config/constants.ts) rather than per-request parameters.

## Frequently Asked Questions

### Why does my OmniRoute stream close with a gateway timeout even though the LLM is still processing?

The `createSSEStream` function enforces `STREAM_IDLE_TIMEOUT_MS` (default ~30 seconds) in lines 591‑620 of [`open-sse/utils/stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/utils/stream.ts). If the upstream provider enters a long reasoning phase without emitting SSE data, the watchdog terminates the connection. Increase the `STREAM_IDLE_TIMEOUT_MS` environment variable or implement a keep-alive mechanism at the application layer to prevent premature closure.

### How can I prevent OmniRoute from dropping the `[DONE]` terminator for non-OpenAI clients?

You cannot override this behavior per-request without modifying the source code. Lines 444‑447 in [`stream.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/stream.ts) explicitly suppress the `[DONE]` line for translated streams. You must modify the client to detect stream completion via connection closure or provider-specific end events (like `message_stop` for Anthropic), as shown in the fetch example above.

### What causes tool-call streaming to fail or produce malformed JSON in OmniRoute?

The `applyTextualToolCallStreamingGuard` parser (lines 554‑580) only scans the `content` field of `delta` objects for simple single-line tool calls. Complex multi-line function arguments or non-standard payload structures bypass the regex detection, causing the stream to emit incomplete JSON. Pre-format tool calls as single-line strings or post-process the stream client-side to reconstruct fragmented tool-call chunks.

### Why does OmniRoute inject synthetic empty responses for Claude models?

When Anthropic's API returns no content blocks, the `emitSyntheticClaudeEmptyResponse` utility (lines 813‑847) generates a placeholder to prevent client-side parsing errors. This behavior can be disabled by setting `suppressThinkClose: true` in the `StreamOptions` passed to `createSSEStream`, though this may expose downstream parsers to empty delta objects.