How OmniRoute's OpenAI-Compatible Endpoint Works: Complete Technical Deep-Dive

OmniRoute's /v1/chat/completions endpoint implements a layered request pipeline that validates incoming OpenAI‑style payloads, routes them through provider‑specific translators, and streams responses back using Server‑Sent Events.

The OmniRoute project (diegosouzapw/OmniRoute) provides a unified gateway for multiple LLM providers through a single OpenAI‑compatible HTTP API. This architecture lets developers use standard OpenAI SDKs while transparently accessing Anthropic Claude, Azure OpenAI, local models, and more.

API Route: The Entry Point

The HTTP surface is defined in Next.js App Router at src/app/api/v1/chat/completions/route.ts.

The exported POST handler executes this validation chain:

• CORS pre-flight — enables cross‑origin browser calls
• Zod validation — enforces the ChatCompletionRequest schema against OpenAI's specification
• Authentication extraction — parses API keys via extractApiKey() and validates via validateApiKey()
• Policy enforcement — applies rate limits, quotas, and safety guardrails before forwarding

Once validated, the handler delegates to the core streaming engine.

Core Streaming Handler

The orchestration logic lives in open-sse/handlers/chatCore.ts.

The handleChatCore() function:

1. Decorates the payload with internal metadata (request ID, timestamps, tracing context)
2. Invokes the combo routing subsystem to select candidate providers
3. Manages the request lifecycle through translation, execution, and response streaming

This handler is provider‑agnostic—all provider specifics are abstracted into later pipeline stages.

Combo Routing and Provider Selection

OmniRoute implements sophisticated routing through open-sse/services/combo.ts.

The combo service evaluates routing strategies:

Strategy	Behavior
auto (default)	Intelligent selection based on model availability and health
weighted	Distribution across providers by configured weights
round-robin	Cyclic distribution across healthy candidates

Before dispatch, the pipeline applies resilience patterns from src/shared/utils/circuitBreaker.ts and src/sse/services/accountFallback.ts:

• Provider‑level circuit breakers track error rates and disable failing endpoints
• Connection‑level cooldowns throttle individual API keys hitting rate limits

Request Translation Layer

OmniRoute transforms OpenAI‑shaped payloads into provider‑native formats through open-sse/translator/index.ts.

The translateRequest() function dispatches to provider‑specific modules:

• open-sse/translator/anthropic.ts — OpenAI → Anthropic message format
• open-sse/translator/azure.ts — OpenAI → Azure OpenAI request shape
• open-sse/translator/local.ts — OpenAI → local model formats (Ollama, vLLM, etc.)

Each translator handles parameter mapping, message role conversion, and provider‑specific constraints like max_tokens boundaries.

Executor: Upstream HTTP Calls

For each routing candidate, the executor layer sends the transformed request to the upstream LLM service.

Base implementation: open-sse/executors/baseExecutor.ts

Provider subclasses in the same directory implement:

• Retry logic with exponential backoff
• Header propagation (including upstream rate‑limit indicators)
• Streaming response handling via fetch/Response.body readers

The executor returns a raw byte stream regardless of provider response format.

Response Translation and SSE Streaming

The transformer layer at open-sse/transformer/responseTransformer.ts converts upstream responses back to OpenAI‑compatible format.

For streaming responses (stream: true):

• Emits Server‑Sent Events with event: answer lines
• Each chunk contains ChatCompletionResponse shaped JSON
• Preserves delta content format for token‑by‑token delivery

For non‑streaming responses (stream: false):

• Buffers the complete upstream response
• Returns single JSON object matching OpenAI's completion schema

Error Handling and Security

All errors funnel through open-sse/utils/error.ts.

The buildErrorBody() and sanitizeErrorMessage() functions guarantee:

• No raw stack traces leak to callers
• HTTP status codes map to OpenAI‑compatible error shapes
• Provider‑specific errors are masked behind generic messages when configured

This satisfies enterprise security policies while maintaining SDK compatibility.

Usage Examples

Basic curl Request

curl https://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -d '{
        "model": "gpt-4o-mini",
        "messages": [{ "role": "user", "content": "Explain quantum entanglement in plain English." }],
        "max_tokens": 500,
        "stream": true
      }'

Official OpenAI SDK (Node.js)

import { OpenAI } from "openai";

const client = new OpenAI({
  baseURL: "http://localhost:20128/v1",
  apiKey: process.env.OMNIRoute_API_KEY,
});

const stream = await client.chat.completions.create({
  model: "gpt-4o-mini",
  messages: [{ role: "user", content": "What is the capital of Brazil?" }],
  stream: true,
});

for await (const chunk of stream) {
  console.log(chunk.choices[0].delta?.content || "");
}

Routing to Non‑OpenAI Providers

curl https://localhost:20128/v1/chat/completions \
  -H "Authorization: Bearer $OMNIRoute_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "anthropic/claude-3-5-sonnet",
        "messages": [{ "role": "user", "content": "Write a haiku about autumn." }],
        "max_tokens": 100
      }'

The model identifier prefix (anthropic/, azure/, local/) triggers automatic provider selection and request translation.

Key Source Files

Component	File Path	Responsibility
API entry	src/app/api/v1/chat/completions/route.ts	HTTP handling, validation, auth delegation
Core orchestration	open-sse/handlers/chatCore.ts	Request lifecycle, metadata, routing coordination
Routing logic	open-sse/services/combo.ts	Strategy evaluation, candidate selection
Circuit breaker	src/shared/utils/circuitBreaker.ts	Provider health tracking, failure isolation
Request translation	open-sse/translator/index.ts	OpenAI-to-provider payload transformation
HTTP execution	open-sse/executors/baseExecutor.ts (and subclasses)	Upstream LLM communication, retries, streaming
Response formatting	open-sse/transformer/responseTransformer.ts	Provider-to-OpenAI response normalization
Error sanitization	open-sse/utils/error.ts	Secure error payload construction

Summary

OmniRoute's OpenAI‑compatible endpoint works through these architectural principles:

• Single HTTP surface — drop‑in replacement for OpenAI's /v1/chat/completions route
• Pluggable provider stack — translators and executors isolate provider specifics
• Intelligent routing — combo strategies with circuit‑breaker resilience
• Bidirectional transformation — OpenAI schema ↔ native provider formats
• Standards‑compliant streaming — SSE output compatible with official SDKs

Frequently Asked Questions

How does OmniRoute handle authentication for different providers?

OmniRoute extracts the caller's API key via extractApiKey() in the route handler, then maps that key to provider‑specific credentials stored in its configuration. Each executor subclass retrieves the appropriate upstream key when calling the target LLM service, so callers use one OmniRoute key while the gateway manages multiple provider credentials internally.

Can I force a specific provider instead of using auto‑routing?

Yes. Prefix your model identifier with the provider namespace: anthropic/claude-3-opus, azure/gpt-4, or local/llama3.1:70b. The combo service in open-sse/services/combo.ts parses this prefix and bypasses automatic selection, routing directly to the specified provider's executor.

What happens when a provider fails mid‑stream?

The circuit‑breaker in src/shared/utils/circuitBreaker.ts tracks failure rates per provider. When thresholds are exceeded, the provider enters an open state and subsequent requests exclude it from candidate lists. For in‑flight streams, the executor's retry logic attempts the next candidate from the combo routing list, maintaining SSE connection continuity when possible.

Is the streaming format identical to OpenAI's?

Yes. The response transformer at open-sse/transformer/responseTransformer.ts emits data: lines containing JSON matching OpenAI's ChatCompletionStreamResponse shape, including id, object, created, model, and choices[].delta fields. Clients using the official openai Python or Node.js SDKs work without modification.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}