OmniRoute Request Pipeline Flow: From API Route to Upstream Provider

OmniRoute processes client requests through a 10-stage layered pipeline that handles admission control, guardrails, combo routing, and provider execution before returning an OpenAI-compatible response.

Understanding the request pipeline flow in OmniRoute is essential for debugging routing issues, optimizing latency, and extending the proxy's capabilities. This Next.js-based LLM gateway implements a modular architecture where each stage is isolated in dedicated source files, enabling flexible provider failover and policy enforcement.

Stage 1: API Entry Point and CORS Handling

Every request enters through Next.js API routes under src/app/api/v1/…. For chat completions, the entry point is src/app/api/v1/chat/completions/route.ts.

This route performs three critical functions:

• CORS preflight handling for OPTIONS requests
• JSON shape validation to reject malformed payloads early
• Streaming detection based on Accept headers or stream: true in the body

The route then hands the request to handleChat from open-sse/handlers/chat.ts. Entry point logic appears around lines 81-106 of route.ts.

Stage 2: Admission Queue and Throttling

handleChat wraps execution in an admission queue that limits concurrent heavyweight work. This prevents resource exhaustion under load.

Key checks include:

• Request body validation
• Back-pressure detection
• API-key and session limit enforcement

The admission controller is implemented via chatAdmission.withChatAdmission() in open-sse/handlers/chat.ts.

Stage 3: Prompt Injection Guard

A singleton createInjectionGuard() inspects the parsed JSON payload for suspicious patterns before any upstream call occurs. This security layer runs at lines 28-35 of chat.ts and can block requests returning HTTP 400 with injection-specific error codes.

Stage 4: Early Stream Keep-Alive Determination

When the client requests SSE streaming—detected via Accept: text/event-stream or stream: true—the pipeline wraps the handler with withEarlyStreamKeepalive. This emits keep-alive frames while the upstream call is pending, preventing client timeouts during slow provider responses.

Configuration options for keep-alive intervals are passed at lines 91-101 of chat.ts.

Stage 5: Core Chat Implementation

handleChatImplementation performs the bulk of business logic starting around line 47 of chat.ts:

1. Single-parse body resolution via resolveChatRequestBody
2. Reasoning parameter normalization for thinking/reasoning models
3. Routing model resolution via resolveRoutingModel
4. Alias expansion, custom connector discovery, and no-thinking shortcuts
5. Guardrail verification and API-key policy checks
6. Session handling for stateful conversations

This stage prepares the canonical request object used throughout the remaining pipeline.

Stage 6: Combo vs. Single-Model Routing Decision

Using getComboForModel, the system determines whether the target is a combo (multiple fallback models with a defined strategy) or a single model.

• Combo found → dispatch to handleComboChat
• Single model → proceed to handleSingleModelChat

This branching logic appears at lines 72-89 of chat.ts. The combo engine supports 19 routing strategies including priority, weighted, and auto-selection.

Stage 7: Combo Routing Execution

handleComboChat iterates over the combo's target models, invoking handleSingleModelChat for each candidate. It implements:

• Strategy-specific ordering and selection
• Per-target credential pre-fetch via getProviderCredentialsWithQuotaPreflight
• Telemetry recording for each attempt
• Failover logic when upstream providers error

Combo implementation spans lines 120-165 of chat.ts.

Stage 8: Single-Model Execution Pipeline

handleSingleModelChat resolves the concrete provider and model through three phases:

Model Resolution

resolveModelOrError in open-sse/handlers/chatHelpers.ts transforms a model string into validated provider and model identifiers.

Credential Selection

getProviderCredentialsWithQuotaPreflight selects a usable credential bundle with quota headroom, considering:

• Account priority and routing tiers
• Rate limit remaining
• Circuit-breaker state

Circuit-Breaker and Execution

executeChatWithBreaker applies provider-level circuit-breaker logic before invoking the executor. The breaker is managed in open-sse/services/accountFallback.ts.

Stage 9: Provider HTTP Request and Response Translation

Concrete executors in open-sse/executors/ extend BaseExecutor from open-sse/executors/base.ts. Each executor:

• Builds provider-specific HTTP headers and payload structure
• Handles provider streaming quirks
• Applies timeout and abort signal handling

// From open-sse/executors/base.ts (lines 1-30)
export abstract class BaseExecutor {
  async execute(opts: ExecutorOptions): Promise<Response> {
    const { endpoint, headers, body, signal } = opts;
    const response = await fetch(endpoint, {
      method: 'POST',
      headers,
      body: JSON.stringify(body),
      signal,
    });
    return response;
  }
}

Provider-specific implementations include:

• open-sse/executors/azure-openai.ts – Azure OpenAI Service
• open-sse/executors/gemini-web.ts – Google Gemini
• Additional executors for Anthropic, Groq, and custom connectors

Responses are wrapped in a request-telemetry object, then normalized to OpenAI-compatible format by open-sse/translator/index.ts.

Stage 10: Response Finalization and Client Delivery

The pipeline decorates the final response with:

• X-Correlation-Id for distributed tracing
• Session continuity headers
• Modality-bridge headers for multi-modal responses
• Compression-echo when Accept-Encoding was present

For streaming requests, the early-keepalive wrapper is removed and SSE frames flow directly to the client. Non-streaming responses return complete JSON objects.

Final response assembly appears at lines 225-250 of chat.ts.

Complete Pipeline Code Example

The following snippet from src/app/api/v1/chat/completions/route.ts demonstrates the entry-point orchestration:

export async function POST(request) {
  await ensureInitialized();               // init translators once
  
  // CORS and Content-Type validation...
  
  const parsedBody = await request.json();
  const { blocked, result } = injectionGuard(parsedBody);
  if (blocked) return /* 400 injection block */;
  
  const wantsStreaming = (parsedBody?.stream === true) ||
    acceptHeaderForcesStream(request.headers.get('accept'), parsedBody.stream);
  
  if (wantsStreaming) {
    const handlerResponse = releaseChatAdmissionAfterHandler(
      handleChat(request, null, parsedBody, generateRequestId()),
      admission.lease
    );
    const streamedResponse = await withEarlyStreamKeepalive(handlerResponse, { … });
    return withCompressionHeaderEcho(streamedResponse, compressionRequestHeader);
  }
  return /* non-streaming response */;
}

And the core handler dispatch from open-sse/handlers/chat.ts:

export const handleChat = chatAdmission.withChatAdmission(async (request, preParsedBody) => {
  const body = await resolveChatRequestBody(request, preParsedBody);
  
  const { modelStr } = resolveRoutingModel(request, body);
  const combo = await getComboForModel(modelStr);
  
  if (combo) {
    return handleComboChat({ body, combo, … });
  }
  return handleSingleModelChat(body, modelStr, …);
});

Key Source Files Reference

Functionality	File Path
API route entry	src/app/api/v1/chat/completions/route.ts
Core handler and pipeline	open-sse/handlers/chat.ts
Model/provider resolution	open-sse/handlers/chatHelpers.ts
Credential and quota selection	open-sse/services/auth.ts
Circuit-breaker logic	open-sse/services/accountFallback.ts
Executor base class	open-sse/executors/base.ts
Response translation	open-sse/translator/index.ts
Combo routing strategies	open-sse/services/combo.ts

Summary

• OmniRoute's request pipeline consists of 10 sequential stages from API entry to upstream provider
• Admission control prevents resource exhaustion before any provider call begins
• Guardrails and injection detection run early to reject malicious payloads
• Combo routing enables sophisticated failover across multiple providers and models
• Circuit-breaker logic isolates failing providers automatically
• Executable base classes allow clean extension for new LLM providers
• Translator layer ensures consistent OpenAI-compatible responses regardless of upstream format

Frequently Asked Questions

How does OmniRoute handle streaming vs. non-streaming requests?

The pipeline detects streaming intent via the Accept header or stream: true in the request body. Streaming requests are wrapped with withEarlyStreamKeepalive to prevent client timeouts, while non-streaming requests proceed through the standard JSON response path. Both paths share the same core execution logic through handleChat.

What happens when all providers in a combo fail?

handleComboChat iterates through combo targets until success or exhaustion. If all targets fail, the final error response propagates to the client with detailed telemetry about each attempt. The combo engine in open-sse/services/combo.ts supports 19 strategies that control ordering, weighting, and retry behavior.

Where is the circuit-breaker state maintained?

Circuit-breaker state lives in open-sse/services/accountFallback.ts, which tracks provider health, cooldown periods, and model-level lockouts. The breaker is checked in executeChatWithBreaker before any HTTP request is dispatched, with automatic recovery based on configured timeouts and success thresholds.

Can custom guardrails be added to the pipeline?

Yes. The guardrail system in src/lib/guardrails/ supports custom pre-call hooks that run after body parsing but before model resolution. The prompt-injection guard demonstrates this pattern, and additional guardrails can be registered through the same singleton pattern used by createInjectionGuard().

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}