How to Set Up OmniRoute for Streaming: Server-Sent Events Configuration

OmniRoute delivers AI-generated outputs as Server-Sent Events (SSE) when clients set stream: true in the JSON body or send an Accept: text/event-stream header, orchestrated through the request handler in src/app/api/v1/chat/completions/route.ts and the streaming pipeline in open-sse/handlers/chat.ts.

OmniRoute is an open-source AI gateway that provides OpenAI-compatible streaming endpoints. When you set up OmniRoute for streaming, the system detects streaming intent via request parameters or headers, then pipes provider responses through a specialized pipeline that maintains low-latency delivery to downstream clients.

Understanding OmniRoute Streaming Architecture

The streaming implementation follows a four-stage pipeline that transforms upstream provider responses into compliant SSE streams.

Request Detection Logic

The entry point at src/app/api/v1/chat/completions/route.ts evaluates the wantsStreaming boolean by checking two conditions:

const wantsStreaming =
  (parsedBodyIsRecord && parsedBody.stream === true) || // explicit true
  acceptForcesStream;                                   // Accept header forces SSE

The acceptForcesStream variable, implemented in open-sse/handlers/chat.ts, treats Accept: text/event-stream as an implicit stream: true when the request body omits the flag.

Core Pipeline Assembly

The open-sse/handlers/chat.ts file contains the streaming orchestration logic. It implements resolveStreamFlag to determine the final streaming mode and executes assembleStreamingPipeline to construct the processing chain. This handler manages keep-alive pings and translates provider-specific errors into standardized SSE events.

Provider Execution Layer

Provider-specific executors in the open-sse/executors/ directory (such as default.ts) forward requests to upstream APIs with stream: true enabled. These executors return a ReadableStream from the provider's HTTP response, which preserves the byte-stream nature of the data without buffering.

Stream Finalization

The src/app/api/v1/relay/chat/completions/streamFinalizer.ts file bridges the upstream ReadableStream to the HTTP response. Its finalizeReadableStream function converts raw provider bytes into properly formatted SSE messages:

const stream = finalizeReadableStream(upstream.body, (error) => { … });
return new Response(stream, { status: upstream.status, headers });

Environment Configuration Variables

Several environment variables control streaming behavior and safety limits:

Variable	Default	Effect
OMNIROUTE_MAX_NONSTREAMING_RESPONSE_BYTES	67108864 (64 MiB)	Hard-cap for buffered non-streaming responses before rejection
STREAM_READINESS_MAX_TIMEOUT_MS	180000 ms	Maximum wait time for the first upstream token before the stream is considered ready
BIFROST_STREAMING_ENABLED	true	Enables streaming on the optional Bifrost sidecar route
OMNIROUTE_AGENT_GOAL_STREAM_RECOVERY	true	Adds early-stream recovery mechanisms for special /goal workflows
OMNIROUTE_STREAMING_DEFAULT	true	Global default for streaming; set to "false" to force JSON responses even when clients request SSE

Configure these in your .env file after cloning the repository.

Step-by-Step Setup Guide

Follow these steps to configure OmniRoute for streaming responses.

1. Install Dependencies

git clone https://github.com/diegosouzapw/OmniRoute.git
cd OmniRoute
npm ci

2. Configure Environment Variables

Create a .env file from the example template. At minimum, provide an API key for your target provider:

OPENAI_API_KEY=sk-...

To disable streaming globally and force JSON responses regardless of client headers:

OMNIROUTE_STREAMING_DEFAULT=false

3. Start the Server

npm run dev

The server listens on http://localhost:3000 by default.

4. Make Streaming Requests

Using the stream flag:

curl http://localhost:3000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Tell me a joke." }],
    "stream": true
  }'

Using the Accept header only:

curl http://localhost:3000/v1/chat/completions \
  -H "Accept: text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{ "role": "user", "content": "Write a haiku." }]
  }'

Both methods return SSE events formatted as:

data: {"id":"…","choices":[{"delta":{"content":"..."},"index":0,"finish_reason":null}]}

5. Configure the Bifrost Sidecar (Optional)

The Bifrost sidecar proxies downstream providers while preserving streaming semantics. Enable it by setting:

BIFROST_API_KEY=sk-...
BIFROST_BASE_URL=https://api.provider.com
BIFROST_STREAMING_ENABLED=true

Request streaming via the sidecar route:

curl http://localhost:3000/v1/relay/chat/completions/bifrost \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Explain quantum tunneling."}],"stream":true}'

The Bifrost handler in src/app/api/v1/relay/chat/completions/bifrost/route.ts evaluates the wantsStream boolean using both the request body flag and the BIFROST_STREAMING_ENABLED environment variable.

Client Implementation Examples

Node.js Streaming Client

import fetch from "node-fetch";

async function streamChat() {
  const response = await fetch("http://localhost:3000/v1/chat/completions", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      model: "gpt-4o-mini",
      messages: [{ role: "user", content: "List three fruits." }],
      stream: true,
    }),
  });

  if (!response.body) throw new Error("No stream available");

  const reader = response.body.getReader();
  const decoder = new TextDecoder();

  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    process.stdout.write(decoder.decode(value));
  }
}

streamChat().catch(console.error);

Command-Line with Buffering Disabled

curl -N http://localhost:3000/v1/chat/completions \
  -H "Accept: text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"Summarize the plot of Hamlet"}]}'

The -N flag disables curl's output buffering, allowing you to see SSE chunks as they arrive from the server.

Key Source Files Reference

Component	File Path	Purpose
API Entry	src/app/api/v1/chat/completions/route.ts	Detects streaming intent via wantsStreaming and routes to core handlers
Core Handler	open-sse/handlers/chat.ts	Implements resolveStreamFlag and assembleStreamingPipeline for stream orchestration
Stream Finalizer	src/app/api/v1/relay/chat/completions/streamFinalizer.ts	Converts ReadableStream to SSE responses via finalizeReadableStream
Bifrost Route	src/app/api/v1/relay/chat/completions/bifrost/route.ts	Sidecar proxy that respects BIFROST_STREAMING_ENABLED
Transform Utilities	open-sse/transformer/stream.ts	Low-level TransformStream that normalizes provider events to OpenAI-compatible chunks
Configuration	docs/reference/ENVIRONMENT.md	Documents streaming-related environment variables

Summary

• OmniRoute streams responses via Server-Sent Events (SSE) when it detects stream: true in the request body or Accept: text/event-stream in headers.
• The streaming pipeline runs through src/app/api/v1/chat/completions/route.ts → open-sse/handlers/chat.ts → executor → streamFinalizer.ts.
• Environment variables like STREAM_READINESS_MAX_TIMEOUT_MS and BIFROST_STREAMING_ENABLED fine-tune timeout behavior and sidecar capabilities.
• The Bifrost sidecar in src/app/api/v1/relay/chat/completions/bifrost/route.ts provides additional proxy functionality while maintaining streaming semantics.
• No additional code changes are required to enable streaming; simply configure your API keys and set the appropriate request flags.

Frequently Asked Questions

How does OmniRoute detect that a client wants streaming?

OmniRoute evaluates the wantsStreaming boolean in src/app/api/v1/chat/completions/route.ts, which returns true if the JSON body contains "stream": true or if the Accept header equals text/event-stream. The latter triggers the acceptForcesStream logic in open-sse/handlers/chat.ts, allowing header-only streaming negotiation.

Can I disable streaming globally for all clients?

Yes. Set OMNIROUTE_STREAMING_DEFAULT=false in your environment configuration. This forces JSON responses even when clients send stream: true or Accept: text/event-stream, effectively overriding client preferences for scenarios where buffering is required.

What is the purpose of the Bifrost sidecar in streaming?

The Bifrost sidecar acts as a reverse-proxy for downstream providers, exposed at /v1/relay/chat/completions/bifrost. When BIFROST_STREAMING_ENABLED is true, it passes the upstream ReadableStream through finalizeReadableStream to preserve SSE formatting, allowing you to chain providers while maintaining real-time output delivery.

How does OmniRoute handle streaming timeouts?

The system uses STREAM_READINESS_MAX_TIMEOUT_MS (default 180,000 ms) to define the maximum wait time for the first meaningful token from the upstream provider. If the provider fails to send data within this window, the stream is considered failed and appropriate error handling is triggered according to the pipeline configuration in open-sse/handlers/chat.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}