# How the Request Pipeline Flows from API Route to Upstream Provider in OmniRoute

> Uncover the 7-stage request pipeline in OmniRoute. Learn how LLM requests flow from API routes, through validation and translation, to upstream providers and back to clients.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: architecture
- Published: 2026-07-26

---

**OmniRoute processes incoming LLM requests through a seven-stage pipeline that starts at a Next.js API route, validates security and capacity, translates formats, executes against upstream providers, and streams the response back to the client.**

OmniRoute is an open-source LLM gateway that normalizes access to multiple AI providers through a single OpenAI-compatible API. The architecture deliberately isolates concerns—CORS handling, admission control, prompt injection defense, format translation, and execution—into discrete, testable layers. This design allows operators to route requests to providers like Anthropic, Gemini, or OpenAI while maintaining consistent request handling and observability.

## Stage 1: API Route Entry and Validation

Every request enters through a Next.js App Router endpoint located at [`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts). This file exports `POST` and `OPTIONS` handlers that enforce strict protocol compliance before any upstream communication occurs.

### CORS and Content-Type Guards

The route immediately handles CORS pre-flight responses via `handleCorsOptions` to support browser-based clients. It then validates the **Content-Type** header, rejecting any non-JSON payloads with HTTP 415.

### Admission and Security Controls

Before parsing the request body, the pipeline invokes `admitChatRequest` (and `admitChatStructure`) to atomically reserve capacity and enforce payload size limits. This admission control prevents resource exhaustion attacks. Next, a **prompt-injection guard** scans the parsed body for malicious patterns; if flagged, the pipeline returns HTTP 400 immediately.

### Streaming Detection

The route determines output mode by checking for `stream: true` in the JSON body or an `Accept: text/event-stream` header. This decision affects whether the pipeline applies SSE wrapping later.

## Stage 2: Translator Initialization

The first request triggers `initTranslators`, a singleton promise that loads all provider-specific translation modules from [`open-sse/translator/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/translator/index.ts). These translators maintain bidirectional mappings between OpenAI's schema and provider-native formats (Anthropic Messages API, Gemini Content API, etc.).

## Stage 3: Core Handler Orchestration

After validation, control passes to `handleChat` in [`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts), which wraps the heavier `handleChatCore` function.

### Combo Resolution

`handleChatCore` calls the combo engine ([`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts)) to expand abstract combo definitions into an ordered list of concrete targets, specifying provider ID, model name, and account credentials. This enables automatic fallback across multiple providers.

### Translation and Executor Selection

For each target, the pipeline runs `translateRequest` to convert the OpenAI-style payload into provider-specific JSON. It then selects an executor via `getExecutor` ([`open-sse/executors/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/index.ts)). Most providers use the `DefaultExecutor`, though OAuth-based providers receive specialized implementations.

## Stage 4: Executor Dispatch and Upstream Communication

The executor ([`open-sse/executors/default.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/default.ts)) constructs the upstream request through three sequential steps:

1. **URL Construction**: `buildUrl` assembles the provider endpoint using the model and account configuration.
2. **Header Assembly**: `buildHeaders` merges authentication credentials (API keys or OAuth tokens) with required content-type markers.
3. **Request Transformation**: `transformRequest` applies final provider-specific mutations to the payload.

The executor performs the HTTP fetch with exponential backoff retry logic, streams the raw response back to the core handler, and surfaces upstream errors as standardized OmniRoute error objects.

## Stage 5: Response Translation and Streaming

Raw provider responses undergo **response translation** to map provider-specific fields back to the OpenAI-compatible schema expected by clients.

### SSE Streaming and Keep-Alive

For streaming requests, the pipeline wraps the response with `withEarlyStreamKeepalive` (located in the route file). This transform emits periodic `data: {"type":"ping"}` frames to prevent connection timeouts during long-generation tasks. If the client used the `/responses` endpoint, the stream passes through [`responsesTransformer.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/responsesTransformer.ts) to convert Chat Completions chunks into Responses API events.

## Complete Pipeline Examples

### Non-Streaming JSON Request

```http
POST /api/v1/chat/completions HTTP/1.1
Host: localhost:3000
Content-Type: application/json
Accept: application/json

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Tell me a joke." }],
  "stream": false
}

```

The pipeline validates headers, checks admission quotas, runs the injection guard, translates the payload to the provider format, executes via `DefaultExecutor`, and returns a single JSON body.

### Streaming SSE Request

```http
POST /api/v1/chat/completions HTTP/1.1
Host: localhost:3000
Content-Type: application/json
Accept: text/event-stream

{
  "model": "claude-2.1",
  "messages": [{ "role": "user", "content": "Write a short story." }],
  "stream": true
}

```

Because `stream: true` is set, the route activates `withEarlyStreamKeepalive`, emitting `[OPENAI_KEEPALIVE_FRAME]` pings between token chunks until the upstream provider closes the connection.

### Combo Routing with Fallback

```http
POST /api/v1/combo/12345/chat/completions HTTP/1.1
Content-Type: application/json
Accept: application/json

{
  "model": "auto",
  "messages": [{ "role": "user", "content": "Summarize this article." }],
  "stream": false
}

```

The combo resolver expands ID `12345` into an ordered list of targets; `handleChatCore` iterates through them until one succeeds, providing automatic failover across providers.

## Key Files Reference

- **[`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts)**: Entry point handling CORS, admission (`admitChatRequest`), injection guards, and streaming detection.
- **[`open-sse/handlers/chatCore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/handlers/chatCore.ts)**: Core orchestration logic (`handleChatCore`) managing combo resolution and executor dispatch.
- **[`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts)**: Combo routing engine expanding abstract combos into concrete provider/model targets.
- **[`open-sse/translator/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/translator/index.ts)**: Request/response translation layer between OpenAI and provider formats.
- **[`open-sse/executors/index.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/index.ts)**: Executor factory (`getExecutor`) selecting appropriate execution strategies.
- **[`open-sse/executors/default.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/default.ts)**: Default executor implementing fetch logic with exponential backoff and error normalization.
- **[`src/middleware/promptInjectionGuard.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/middleware/promptInjectionGuard.ts)**: Security middleware scanning for prompt injection attacks.
- **[`src/shared/middleware/chatBodyAdmission.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/shared/middleware/chatBodyAdmission.ts)**: Capacity and size enforcement before JSON parsing.

## Summary

- **Entry validation** occurs at [`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts), enforcing CORS, content-type, admission control, and prompt injection checks.
- **Format translation** happens through a singleton-loaded translator system that maps OpenAI schemas to provider-native formats.
- **Execution** is handled by provider-specific executors, typically `DefaultExecutor`, which manage upstream URLs, headers, and retry logic.
- **Combo routing** enables automatic failover by expanding combo definitions into ordered provider targets in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts).
- **Streaming responses** include keep-alive frames via `withEarlyStreamKeepalive` to maintain SSE connections during long generations.

## Frequently Asked Questions

### How does OmniRoute handle streaming versus non-streaming requests?

The pipeline checks for `stream: true` in the request body or an `Accept: text/event-stream` header at the API route level. For non-streaming requests, the executor returns the complete translated JSON; for streaming requests, the response passes through `withEarlyStreamKeepalive` ([`src/app/api/v1/chat/completions/route.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/app/api/v1/chat/completions/route.ts)) to emit periodic ping frames and prevent gateway timeouts.

### What security measures are applied before a request reaches the upstream provider?

OmniRoute applies three layers of defense: **admission control** (`admitChatRequest`) enforces payload size limits and capacity reservations; **prompt-injection detection** ([`src/middleware/promptInjectionGuard.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/middleware/promptInjectionGuard.ts)) scans for malicious patterns; and **content-type validation** rejects non-JSON bodies with HTTP 415 before parsing occurs.

### How does the combo routing system work for provider fallback?

The combo resolver in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) expands a combo ID into an ordered list of concrete targets (provider + model + account). The `handleChatCore` function iterates through these targets sequentially; if one provider fails or times out, the pipeline automatically retries with the next target in the list until success or exhaustion.

### What happens if the upstream provider returns an error?

The `DefaultExecutor` ([`open-sse/executors/default.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/executors/default.ts)) applies exponential backoff retry logic for transient failures. For non-retryable errors (HTTP 4xx or final retry exhaustion), the executor surfaces the error as a standardized OmniRoute error object with the upstream status code and message, which the API route returns to the client without exposing internal stack traces.