How OmniRoute Structures Its OpenAI-Compatible API Across Endpoints

OmniRoute exposes a unified OpenAI-compatible REST API under /api/v1/ using Next.js App Router, with each endpoint delegating to modular handlers in open-sse/handlers/ after CORS guards, Zod validation, and policy checks.

This architecture lets developers call 300+ LLM providers through a single interface that mirrors OpenAI's official surface. The diegosouzapw/OmniRoute repository implements this through five pipeline stages: method guarding, request validation, authentication, handler delegation, and provider-specific execution.

Core Pipeline Architecture

Every OpenAI-compatible route in OmniRoute follows the same execution flow. Understanding this pipeline explains how the system maintains consistency across diverse endpoints.

Stage 1: CORS and Method Guarding

Route files export GET or POST handlers that Next.js wires automatically. Each handler begins with CORS headers and HTTP method validation before processing continues.

Stage 2: Zod Schema Validation

Request bodies and query parameters undergo strict validation using Zod schemas. This catches malformed payloads early and returns standardized error responses matching OpenAI's format.

Stage 3: Authentication and Policy Enforcement

The system extracts API keys via extractApiKey, then applies layered policies:

• Rate limiting per key and per provider
• Prompt injection detection guards
• Circuit-breaker health checks against known-failing providers

Stage 4: Handler Delegation

Validated requests forward to core handlers in open-sse/handlers/:

• chatCore for chat completions
• completionCore for legacy text completions
• embeddingsCore for vector generation
• imagesCore for DALL-E-style generation
• audioCore for Whisper transcription/translation
• modelsCore for catalogue queries

Stage 5: Translation and Execution

Handlers translate OpenAI payloads to provider-specific formats using translators in open-sse/translator/, select executors from open-sse/executors/, and stream responses back via Server-Sent Events (SSE) when stream: true.

Endpoint Reference: File Paths and Responsibilities

OpenAI Concept	Route File	Core Handler	Purpose
Chat Completions	src/app/api/v1/chat/completions/route.ts	open-sse/handlers/chat.ts	Main streaming chat endpoint
Text Completions (legacy)	src/app/api/v1/completions/route.ts	open-sse/handlers/completions.ts	Non-chat completion API
Embeddings	src/app/api/v1/embeddings/route.ts	open-sse/handlers/embeddings.ts	Vector representation generation
Image Generations	src/app/api/v1/images/generations/route.ts	open-sse/handlers/images.ts	DALL-E-compatible image creation
Audio Transcriptions	src/app/api/v1/audio/transcriptions/route.ts	open-sse/handlers/audio.ts	Speech-to-text (Whisper)
Audio Translations	src/app/api/v1/audio/translations/route.ts	open-sse/handlers/audio.ts	Cross-language transcription
Audio Speech	src/app/api/v1/audio/speech/route.ts	open-sse/handlers/audio.ts	Text-to-speech generation
Models List	src/app/api/v1/models/route.ts	open-sse/handlers/models.ts	Available model catalogue
Model Retrieval	src/app/api/v1/models/[...model]/route.ts	open-sse/handlers/models.ts	Single model details

All these routes use open-sse/executors/openaiCompatible.ts for execution unless provider-scoped alternatives apply.

Provider-Scoped and Specialized Endpoints

Forcing Provider Selection

The /v1/providers/{provider}/ namespace bypasses automatic provider selection. Instead of letting OmniRoute choose based on model prefix, these endpoints force routing to a specific backend.

curl -X POST http://localhost:20128/v1/providers/anthropic/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "anthropic/claude-3-5-sonnet",
    "messages": [{"role": "user", "content": "Write a haiku about spring."}]
  }'

The route at src/app/api/v1/providers/[provider]/chat/completions/route.ts pulls provider configuration from open-sse/config/providerRegistry.ts, which defines base URLs, authentication schemes, and format flags (OpenAI, Claude, or Gemini).

VS Code Token Alias

Extensions embedding OmniRoute can pass raw API keys in the URL path:

curl -X POST http://localhost:20128/api/v1/vscode/sk-abcdef12345/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [{"role": "user", "content": "What is the capital of France?"}]
  }'

The handler at src/app/api/v1/vscode/[token]/chat/completions/route.ts extracts the token from the path segment, validates it as a raw key, then proceeds with standard chat handling.

Relay Mode Endpoints

For debugging or custom proxy scenarios, src/app/api/v1/relay/chat/completions/route.ts and src/app/api/v1/relay/bifrost/route.ts provide bypass routes. These skip policy checks already performed upstream, supporting either TypeScript-native relay or the external Bifrost side-car proxy.

Combo Routing and Auto-Selection

The /v1/combos/ and /v1/auto-combo/ endpoints implement intelligent request distribution:

curl -X POST http://localhost:20128/v1/auto-combo/default/candidates \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto",
    "messages": [{"role": "user", "content": "Summarize the plot of Inception."}]
  }'
`

The route `src/app/api/v1/auto-combo/[channel]/candidates/route.ts` delegates to `open-sse/services/combo.ts`, which applies strategies (auto, weighted, fusion) before invoking standard chat handlers with selected model targets.

## Streaming vs. JSON Response Handling

Every endpoint respects the `stream` parameter (default: `true`) and `Accept` header. When streaming is requested, OmniRoute:

1. Applies early keep-alive SSE guards to prevent connection timeouts
2. Uses chunked transfer encoding with `data:` prefixed SSE frames
3. Closes with `[DONE]` marker matching OpenAI's format

For `stream: false` or `Accept: application/json`, the executor buffers the complete response and returns standard JSON.

## Resilience Mechanisms

OmniRoute's OpenAI-compatible API incorporates failure handling at multiple layers:

- **Circuit breaker**: `src/shared/utils/circuitBreaker.ts` tracks provider health and excludes failing backends from selection
- **Account fallback**: `src/sse/services/accountFallback.ts` rotates credentials when rate limits or quota errors occur
- **Provider cooldown**: Automatic backoff for transient errors before re-enabling providers

These mechanisms operate outside route files, keeping endpoint code clean while ensuring high availability.

## Complete Code Examples

### Standard Chat Completion

```bash
curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {"role": "user", "content": "Explain quantum tunneling in one sentence."}
    ],
    "temperature": 0.7
  }'

Embedding Generation

curl -X POST http://localhost:20128/v1/embeddings \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "model": "openai/text-embedding-3-small",
    "input": "The quick brown fox jumps over the lazy dog."
  }'

Image Generation

curl -X POST http://localhost:20128/v1/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-your-key" \
  -d '{
    "prompt": "A futuristic city at sunset, photorealistic",
    "n": 2,
    "size": "1024x1024"
  }'

Audio Transcription

curl -X POST http://localhost:20128/v1/audio/transcriptions \
  -H "Authorization: Bearer sk-your-key" \
  -F file=@speech.wav \
  -F model=whisper-1

Key Files for API Extension

File	Responsibility
open-sse/config/providerRegistry.ts	Central registry of 300+ providers with base URLs and format flags
open-sse/translator/openai-to-claude.ts	Example translator converting OpenAI schemas to Claude format
open-sse/translator/openai-to-gemini.ts	Google Gemini payload conversion
open-sse/executors/openaiCompatible.ts	Generic executor for OpenAI-format providers
open-sse/executors/anthropic.ts	Specialized executor for Anthropic's native API
src/shared/utils/circuitBreaker.ts	Provider health tracking and automatic exclusion

Summary

• Unified surface: All endpoints live under /api/v1/ with OpenAI-compatible request/response formats
• Five-stage pipeline: CORS guards → Zod validation → auth/policy → handler delegation → translation/execution
• Modular handlers: Core logic in open-sse/handlers/ keeps route files thin and consistent
• Provider abstraction: Translators and executors in open-sse/ convert between OpenAI schema and 300+ backend formats
• Forced routing: Provider-scoped endpoints (/providers/{provider}/) bypass automatic selection
• Specialized paths: VS Code aliases, relay modes, and combo routing extend baseline functionality
• Resilience: Circuit breakers and fallback logic operate transparently across all endpoints

Frequently Asked Questions

How does OmniRoute handle non-OpenAI providers like Anthropic or Gemini?

OmniRoute uses a translator layer in open-sse/translator/ to convert incoming OpenAI-format requests to provider-native schemas. For Anthropic, openai-to-claude.ts transforms messages, parameters, and response formats. The executor layer then handles provider-specific authentication and streaming protocols, returning responses that match OpenAI's structure to the client.

Can I disable automatic provider selection and force a specific backend?

Yes. Use the provider-scoped endpoint pattern: /v1/providers/{provider}/chat/completions. The route at src/app/api/v1/providers/[provider]/chat/completions/route.ts extracts the provider slug from the path, validates it against providerRegistry.ts, and forces routing to that backend regardless of model prefix or load-balancing rules.

What happens when a provider fails during request processing?

OmniRoute implements circuit-breaker logic via src/shared/utils/circuitBreaker.ts that tracks error rates per provider. After threshold failures, a provider enters a cooldown period and is excluded from selection. For streaming requests, the system attempts account fallback through src/sse/services/accountFallback.ts to rotate credentials before failing the request to the client.

How do I add a new OpenAI-compatible endpoint to OmniRoute?

Create a new folder structure under src/app/api/v1/ matching the desired path, add a route.ts that exports GET/POST handlers, and follow the five-stage pipeline. Import validation schemas from existing endpoints, delegate to or create a handler in open-sse/handlers/, and register any new provider requirements in open-sse/config/providerRegistry.ts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}