OmniRoute Request Pipeline: Complete Flow from Client to Provider Response

OmniRoute processes every request through a 14-stage modular pipeline that transforms client HTTP calls into provider-specific API requests, applies resilience patterns like circuit breakers and retries, and returns unified responses via JSON or SSE streaming.

The diegosouzapw/OmniRoute repository implements this architecture to support 340+ LLM providers through a layered design where each stage can be extended, swapped, or disabled via feature flags. Understanding this pipeline is essential for debugging routing decisions, optimizing latency, or adding custom middleware.

Entry Point: Next.js API Routes

Every request begins at a Next.js App Router endpoint under src/app/api/v1/….

The primary entry point for chat completions is src/app/api/v1/chat/completions/route.ts, which exports HTTP method handlers for the /v1/chat/completions endpoint. This route performs four responsibilities before delegation:

1. CORS handling — Applies cross-origin policies and pre-flight validation
2. Zod validation — Parses and validates request bodies against strict schemas (e.g., ChatCompletionsSchema)
3. Authentication — Optionally extracts and verifies API keys via extractApiKey() and isValidApiKey()
4. Policy enforcement — Runs quota checks, cost limits, and prompt-injection guardrails

// src/app/api/v1/chat/completions/route.ts
export async function POST(req: Request) {
  const body = await req.json();
  // 1️⃣ CORS & Zod validation
  const parsed = ChatCompletionsSchema.parse(body);
  // 2️⃣ Optional auth
  const apiKey = extractApiKey(req);
  // 3️⃣ Policy & guardrails
  await policyCheck(parsed, apiKey);
  // 4️⃣ Delegate to core handler
  return handleChatCore(parsed, req);
}

Core Handler: Orchestration in chatCore.ts

Once validated, the request enters open-sse/handlers/chatCore.ts — the central orchestrator for the OmniRoute request pipeline. This handler coordinates caching, routing, translation, execution, and response streaming through sequential subsystems.

Cache and Rate-Limit Short-Circuits

The handler first checks:

• Response cache — Returns cached results for identical requests
• Per-connection rate limits — Rejects or throttles exceeding connections

These checks can terminate the pipeline early, avoiding unnecessary upstream calls.

Combo Routing Decision

For requests specifying multiple models or fallback strategies, open-sse/services/combo.ts builds candidate target lists and applies one of 19 routing strategies:

• Priority — Try providers in ranked order
• Weighted — Distribute load by configured weights
• Fill-first — Saturate primary before spilling to secondaries
• Round-robin — Cycle through providers evenly

// open-sse/handlers/chatCore.ts (conceptual flow)
export async function handleChatCore(input, req) {
  // Cache, rate-limit, and combo routing
  const target = await resolveComboTargets(input);
  // Translate to provider format
  const providerReq = translateRequest(input, target.provider);
  // Execute with executor + retry/back-off
  const providerRes = await getExecutor(target.provider).execute(providerReq);
  // Translate back, stream, and sanitize
  return streamResponse(providerRes);
}

Translation Layer: Unified Schema to Provider Format

The open-sse/translator/ directory contains provider-specific translators that convert OmniRoute's unified request schema into target API formats. Each translator handles:

• Request payload transformation (OpenAI-compatible, Claude-compatible, etc.)
• Header and authentication injection
• Parameter mapping and defaults

Execution with Resilience Patterns

Executor Selection and HTTP Transport

The open-sse/executors/ directory provides executor implementations chosen by provider type. Each executor manages the actual HTTP transport to upstream LLM services.

Retry, Back-off, and Circuit Breaking

Before any upstream call, executors consult src/shared/utils/circuitBreaker.ts for provider health state:

• Open circuits — Fast-fail requests to unhealthy providers
• Closed circuits — Allow traffic with timeout and retry logic
• Half-open probes — Gradual recovery testing after cooldown

The executor implements exponential back-off, configurable timeouts, and automatic retry for transient failures.

Response Processing and Streaming Pipeline

Reverse Translation

Upstream responses pass back through the translator layer, normalizing provider-specific formats into OmniRoute's unified response schema.

SSE Streaming Stages

For streaming endpoints, open-sse/handlers/chatCore/* modules process each Server-Sent Event chunk through pipeline stages:

Stage	File	Purpose
Quota sharing	streamingQuotaShare.ts	Enforce per-key and per-model usage limits
Usage statistics	streamingUsageStats.ts	Compute token counts and cost attribution
Guardrails	Pipeline integration with src/lib/guardrails/*	Optional PII masking and moderation
Error sanitization	open-sse/utils/error.ts	buildErrorBody() / sanitizeErrorMessage() prevent stack trace leaks

// Client-side consumption of streaming response
const res = await fetch('http://localhost:20128/v1/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${process.env.OMNIRoute_API_KEY}`
  },
  body: JSON.stringify({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: 'Hello, world!' }],
    stream: true
  })
});

// Read SSE stream
const reader = res.body?.getReader();
while (true) {
  const { done, value } = await reader!.read();
  if (done) break;
  console.log(new TextDecoder().decode(value));
}

Final Response Delivery

The pipeline concludes by returning either:

• JSON response — For synchronous requests, fully assembled and validated
• SSE stream — For streaming requests, with proper termination events and error wrapping

Both paths pass through the originating API route, which handles final CORS headers and HTTP status codes before the response reaches the client.

Key Architectural Files

Component	Path	Responsibility
API route entry	src/app/api/v1/chat/completions/route.ts	HTTP handling, validation, auth, delegation
Core orchestrator	open-sse/handlers/chatCore.ts	Pipeline coordination
Combo routing	open-sse/services/combo.ts	Multi-provider strategy selection
Translators	open-sse/translator/*	Schema conversion
Executors	open-sse/executors/*	HTTP transport to providers
Circuit breaker	src/shared/utils/circuitBreaker.ts	Failure detection and recovery
Streaming stages	open-sse/handlers/chatCore/*	SSE processing and enrichment
Guardrails	src/lib/guardrails/*	Content safety and compliance
Error utilities	open-sse/utils/error.ts	Sanitization and formatting

Summary

• OmniRoute's request pipeline spans 14 modular stages from Next.js API route entry through provider response delivery
• Validation and auth happen at the edge in route handlers using Zod schemas
• Core orchestration in chatCore.ts manages caching, combo routing, translation, and execution
• Resilience patterns include circuit breakers (circuitBreaker.ts), exponential back-off, and provider health tracking
• 19 routing strategies in combo.ts enable sophisticated multi-provider and fallback behaviors
• Streaming pipeline stages enforce quotas, compute usage, apply guardrails, and sanitize errors before SSE emission
• Modular design allows each layer to be extended or disabled via configuration

Frequently Asked Questions

How does OmniRoute handle provider failures during a request?

The executor layer consults src/shared/utils/circuitBreaker.ts before each upstream call. When a provider exceeds failure thresholds, its circuit opens and requests fast-fail or route to alternates. The combo service can automatically fail over to secondary providers based on the configured routing strategy.

Where does request validation occur in the OmniRoute pipeline?

Validation happens at the API route entry point using Zod schemas. The src/app/api/v1/chat/completions/route.ts file parses and validates the body against ChatCompletionsSchema before any downstream processing, ensuring only well-formed requests reach the core handler.

Can individual pipeline stages be disabled or customized?

Yes. The modular architecture in open-sse/handlers/chatCore.ts accepts feature flags and configuration objects that control stage activation. Guardrails, caching, and specific streaming processors can each be enabled or disabled without modifying core pipeline code.

What enables OmniRoute to support 340+ LLM providers?

Three abstractions drive provider coverage: the translator layer (open-sse/translator/*) handles format conversion, the executor layer (open-sse/executors/*) manages transport specifics, and the combo routing service enables flexible targeting across heterogeneous provider pools. New providers require only a translator and executor implementation.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}