# OmniRoute Request Pipeline Flow: From API Route to Upstream Provider

> Explore the OmniRoute request pipeline flow. Discover how requests move from API to upstream providers through a 10-stage layered process for efficient routing and execution.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-08-11

---

**OmniRoute processes client requests through a 10-stage layered pipeline that handles admission control, guardrails, combo routing, and provider execution before returning an OpenAI-compatible response.**

Understanding the **request pipeline flow in OmniRoute** is essential for debugging routing issues, optimizing latency, and extending the proxy's capabilities. This Next.js-based LLM gateway implements a modular architecture where each stage is isolated in dedicated source files, enabling flexible provider failover and policy enforcement.

## Stage 1: API Entry Point and CORS Handling

Every request enters through Next.js API routes under `src/app/api/v1/…`. For chat completions, the entry point is [`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts).

This route performs three critical functions:

- **CORS preflight** handling for OPTIONS requests
- **JSON shape validation** to reject malformed payloads early
- **Streaming detection** based on `Accept` headers or `stream: true` in the body

The route then hands the request to `handleChat` from [`open-sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chat.ts). Entry point logic appears around lines 81-106 of [`route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/route.ts).

## Stage 2: Admission Queue and Throttling

`handleChat` wraps execution in an **admission queue** that limits concurrent heavyweight work. This prevents resource exhaustion under load.

Key checks include:

- Request body validation
- Back-pressure detection
- API-key and session limit enforcement

The admission controller is implemented via `chatAdmission.withChatAdmission()` in [`open-sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chat.ts).

## Stage 3: Prompt Injection Guard

A singleton `createInjectionGuard()` inspects the parsed JSON payload for suspicious patterns before any upstream call occurs. This security layer runs at lines 28-35 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts) and can block requests returning HTTP 400 with injection-specific error codes.

## Stage 4: Early Stream Keep-Alive Determination

When the client requests SSE streaming—detected via `Accept: text/event-stream` or `stream: true`—the pipeline wraps the handler with `withEarlyStreamKeepalive`. This emits keep-alive frames while the upstream call is pending, preventing client timeouts during slow provider responses.

Configuration options for keep-alive intervals are passed at lines 91-101 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts).

## Stage 5: Core Chat Implementation

`handleChatImplementation` performs the bulk of business logic starting around line 47 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts):

1. **Single-parse body resolution** via `resolveChatRequestBody`
2. **Reasoning parameter normalization** for thinking/reasoning models
3. **Routing model resolution** via `resolveRoutingModel`
4. **Alias expansion**, custom connector discovery, and no-thinking shortcuts
5. **Guardrail verification** and API-key policy checks
6. **Session handling** for stateful conversations

This stage prepares the canonical request object used throughout the remaining pipeline.

## Stage 6: Combo vs. Single-Model Routing Decision

Using `getComboForModel`, the system determines whether the target is a **combo** (multiple fallback models with a defined strategy) or a single model.

- **Combo found** → dispatch to `handleComboChat`
- **Single model** → proceed to `handleSingleModelChat`

This branching logic appears at lines 72-89 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts). The combo engine supports 19 routing strategies including priority, weighted, and auto-selection.

## Stage 7: Combo Routing Execution

`handleComboChat` iterates over the combo's target models, invoking `handleSingleModelChat` for each candidate. It implements:

- Strategy-specific ordering and selection
- Per-target credential pre-fetch via `getProviderCredentialsWithQuotaPreflight`
- Telemetry recording for each attempt
- Failover logic when upstream providers error

Combo implementation spans lines 120-165 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts).

## Stage 8: Single-Model Execution Pipeline

`handleSingleModelChat` resolves the concrete provider and model through three phases:

### Model Resolution

`resolveModelOrError` in [`open-sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatHelpers.ts) transforms a model string into validated provider and model identifiers.

### Credential Selection

`getProviderCredentialsWithQuotaPreflight` selects a usable credential bundle with quota headroom, considering:

- Account priority and routing tiers
- Rate limit remaining
- Circuit-breaker state

### Circuit-Breaker and Execution

`executeChatWithBreaker` applies provider-level circuit-breaker logic before invoking the executor. The breaker is managed in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts).

## Stage 9: Provider HTTP Request and Response Translation

Concrete executors in `open-sse/executors/` extend `BaseExecutor` from [`open-sse/executors/base.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/base.ts). Each executor:

- Builds provider-specific HTTP headers and payload structure
- Handles provider streaming quirks
- Applies timeout and abort signal handling

```typescript
// From open-sse/executors/base.ts (lines 1-30)
export abstract class BaseExecutor {
  async execute(opts: ExecutorOptions): Promise<Response> {
    const { endpoint, headers, body, signal } = opts;
    const response = await fetch(endpoint, {
      method: 'POST',
      headers,
      body: JSON.stringify(body),
      signal,
    });
    return response;
  }
}

```

Provider-specific implementations include:

- [`open-sse/executors/azure-openai.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/azure-openai.ts) – Azure OpenAI Service
- [`open-sse/executors/gemini-web.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/gemini-web.ts) – Google Gemini
- Additional executors for Anthropic, Groq, and custom connectors

Responses are wrapped in a **request-telemetry** object, then normalized to OpenAI-compatible format by [`open-sse/translator/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/translator/index.ts).

## Stage 10: Response Finalization and Client Delivery

The pipeline decorates the final response with:

- `X-Correlation-Id` for distributed tracing
- Session continuity headers
- Modality-bridge headers for multi-modal responses
- Compression-echo when `Accept-Encoding` was present

For streaming requests, the early-keepalive wrapper is removed and SSE frames flow directly to the client. Non-streaming responses return complete JSON objects.

Final response assembly appears at lines 225-250 of [`chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/chat.ts).

## Complete Pipeline Code Example

The following snippet from [`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts) demonstrates the entry-point orchestration:

```typescript
export async function POST(request) {
  await ensureInitialized();               // init translators once
  
  // CORS and Content-Type validation...
  
  const parsedBody = await request.json();
  const { blocked, result } = injectionGuard(parsedBody);
  if (blocked) return /* 400 injection block */;
  
  const wantsStreaming = (parsedBody?.stream === true) ||
    acceptHeaderForcesStream(request.headers.get('accept'), parsedBody.stream);
  
  if (wantsStreaming) {
    const handlerResponse = releaseChatAdmissionAfterHandler(
      handleChat(request, null, parsedBody, generateRequestId()),
      admission.lease
    );
    const streamedResponse = await withEarlyStreamKeepalive(handlerResponse, { … });
    return withCompressionHeaderEcho(streamedResponse, compressionRequestHeader);
  }
  return /* non-streaming response */;
}

```

And the core handler dispatch from [`open-sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chat.ts):

```typescript
export const handleChat = chatAdmission.withChatAdmission(async (request, preParsedBody) => {
  const body = await resolveChatRequestBody(request, preParsedBody);
  
  const { modelStr } = resolveRoutingModel(request, body);
  const combo = await getComboForModel(modelStr);
  
  if (combo) {
    return handleComboChat({ body, combo, … });
  }
  return handleSingleModelChat(body, modelStr, …);
});

```

## Key Source Files Reference

| Functionality | File Path |
|-------------|-----------|
| API route entry | [`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts) |
| Core handler and pipeline | [`open-sse/handlers/chat.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chat.ts) |
| Model/provider resolution | [`open-sse/handlers/chatHelpers.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatHelpers.ts) |
| Credential and quota selection | [`open-sse/services/auth.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/auth.ts) |
| Circuit-breaker logic | [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) |
| Executor base class | [`open-sse/executors/base.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/base.ts) |
| Response translation | [`open-sse/translator/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/translator/index.ts) |
| Combo routing strategies | [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) |

## Summary

- **OmniRoute's request pipeline** consists of 10 sequential stages from API entry to upstream provider
- **Admission control** prevents resource exhaustion before any provider call begins
- **Guardrails and injection detection** run early to reject malicious payloads
- **Combo routing** enables sophisticated failover across multiple providers and models
- **Circuit-breaker logic** isolates failing providers automatically
- **Executable base classes** allow clean extension for new LLM providers
- **Translator layer** ensures consistent OpenAI-compatible responses regardless of upstream format

## Frequently Asked Questions

### How does OmniRoute handle streaming vs. non-streaming requests?

The pipeline detects streaming intent via the `Accept` header or `stream: true` in the request body. Streaming requests are wrapped with `withEarlyStreamKeepalive` to prevent client timeouts, while non-streaming requests proceed through the standard JSON response path. Both paths share the same core execution logic through `handleChat`.

### What happens when all providers in a combo fail?

`handleComboChat` iterates through combo targets until success or exhaustion. If all targets fail, the final error response propagates to the client with detailed telemetry about each attempt. The combo engine in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) supports 19 strategies that control ordering, weighting, and retry behavior.

### Where is the circuit-breaker state maintained?

Circuit-breaker state lives in [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), which tracks provider health, cooldown periods, and model-level lockouts. The breaker is checked in `executeChatWithBreaker` before any HTTP request is dispatched, with automatic recovery based on configured timeouts and success thresholds.

### Can custom guardrails be added to the pipeline?

Yes. The guardrail system in `src/lib/guardrails/` supports custom pre-call hooks that run after body parsing but before model resolution. The prompt-injection guard demonstrates this pattern, and additional guardrails can be registered through the same singleton pattern used by `createInjectionGuard()`.