How the Request Pipeline Flows from API Route to Upstream Provider in OmniRoute

OmniRoute processes incoming LLM requests through a seven-stage pipeline that starts at a Next.js API route, validates security and capacity, translates formats, executes against upstream providers, and streams the response back to the client.

OmniRoute is an open-source LLM gateway that normalizes access to multiple AI providers through a single OpenAI-compatible API. The architecture deliberately isolates concerns—CORS handling, admission control, prompt injection defense, format translation, and execution—into discrete, testable layers. This design allows operators to route requests to providers like Anthropic, Gemini, or OpenAI while maintaining consistent request handling and observability.

Stage 1: API Route Entry and Validation

Every request enters through a Next.js App Router endpoint located at src/app/api/v1/chat/completions/route.ts. This file exports POST and OPTIONS handlers that enforce strict protocol compliance before any upstream communication occurs.

CORS and Content-Type Guards

The route immediately handles CORS pre-flight responses via handleCorsOptions to support browser-based clients. It then validates the Content-Type header, rejecting any non-JSON payloads with HTTP 415.

Admission and Security Controls

Before parsing the request body, the pipeline invokes admitChatRequest (and admitChatStructure) to atomically reserve capacity and enforce payload size limits. This admission control prevents resource exhaustion attacks. Next, a prompt-injection guard scans the parsed body for malicious patterns; if flagged, the pipeline returns HTTP 400 immediately.

Streaming Detection

The route determines output mode by checking for stream: true in the JSON body or an Accept: text/event-stream header. This decision affects whether the pipeline applies SSE wrapping later.

Stage 2: Translator Initialization

The first request triggers initTranslators, a singleton promise that loads all provider-specific translation modules from open-sse/translator/index.ts. These translators maintain bidirectional mappings between OpenAI's schema and provider-native formats (Anthropic Messages API, Gemini Content API, etc.).

Stage 3: Core Handler Orchestration

After validation, control passes to handleChat in open-sse/handlers/chatCore.ts, which wraps the heavier handleChatCore function.

Combo Resolution

handleChatCore calls the combo engine (open-sse/services/combo.ts) to expand abstract combo definitions into an ordered list of concrete targets, specifying provider ID, model name, and account credentials. This enables automatic fallback across multiple providers.

Translation and Executor Selection

For each target, the pipeline runs translateRequest to convert the OpenAI-style payload into provider-specific JSON. It then selects an executor via getExecutor (open-sse/executors/index.ts). Most providers use the DefaultExecutor, though OAuth-based providers receive specialized implementations.

Stage 4: Executor Dispatch and Upstream Communication

The executor (open-sse/executors/default.ts) constructs the upstream request through three sequential steps:

1. URL Construction: buildUrl assembles the provider endpoint using the model and account configuration.
2. Header Assembly: buildHeaders merges authentication credentials (API keys or OAuth tokens) with required content-type markers.
3. Request Transformation: transformRequest applies final provider-specific mutations to the payload.

The executor performs the HTTP fetch with exponential backoff retry logic, streams the raw response back to the core handler, and surfaces upstream errors as standardized OmniRoute error objects.

Stage 5: Response Translation and Streaming

Raw provider responses undergo response translation to map provider-specific fields back to the OpenAI-compatible schema expected by clients.

SSE Streaming and Keep-Alive

For streaming requests, the pipeline wraps the response with withEarlyStreamKeepalive (located in the route file). This transform emits periodic data: {"type":"ping"} frames to prevent connection timeouts during long-generation tasks. If the client used the /responses endpoint, the stream passes through responsesTransformer.ts to convert Chat Completions chunks into Responses API events.

Complete Pipeline Examples

Non-Streaming JSON Request

POST /api/v1/chat/completions HTTP/1.1
Host: localhost:3000
Content-Type: application/json
Accept: application/json

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "Tell me a joke." }],
  "stream": false
}

The pipeline validates headers, checks admission quotas, runs the injection guard, translates the payload to the provider format, executes via DefaultExecutor, and returns a single JSON body.

Streaming SSE Request

POST /api/v1/chat/completions HTTP/1.1
Host: localhost:3000
Content-Type: application/json
Accept: text/event-stream

{
  "model": "claude-2.1",
  "messages": [{ "role": "user", "content": "Write a short story." }],
  "stream": true
}

Because stream: true is set, the route activates withEarlyStreamKeepalive, emitting [OPENAI_KEEPALIVE_FRAME] pings between token chunks until the upstream provider closes the connection.

Combo Routing with Fallback

POST /api/v1/combo/12345/chat/completions HTTP/1.1
Content-Type: application/json
Accept: application/json

{
  "model": "auto",
  "messages": [{ "role": "user", "content": "Summarize this article." }],
  "stream": false
}

The combo resolver expands ID 12345 into an ordered list of targets; handleChatCore iterates through them until one succeeds, providing automatic failover across providers.

Key Files Reference

• src/app/api/v1/chat/completions/route.ts: Entry point handling CORS, admission (admitChatRequest), injection guards, and streaming detection.
• open-sse/handlers/chatCore.ts: Core orchestration logic (handleChatCore) managing combo resolution and executor dispatch.
• open-sse/services/combo.ts: Combo routing engine expanding abstract combos into concrete provider/model targets.
• open-sse/translator/index.ts: Request/response translation layer between OpenAI and provider formats.
• open-sse/executors/index.ts: Executor factory (getExecutor) selecting appropriate execution strategies.
• open-sse/executors/default.ts: Default executor implementing fetch logic with exponential backoff and error normalization.
• src/middleware/promptInjectionGuard.ts: Security middleware scanning for prompt injection attacks.
• src/shared/middleware/chatBodyAdmission.ts: Capacity and size enforcement before JSON parsing.

Summary

• Entry validation occurs at src/app/api/v1/chat/completions/route.ts, enforcing CORS, content-type, admission control, and prompt injection checks.
• Format translation happens through a singleton-loaded translator system that maps OpenAI schemas to provider-native formats.
• Execution is handled by provider-specific executors, typically DefaultExecutor, which manage upstream URLs, headers, and retry logic.
• Combo routing enables automatic failover by expanding combo definitions into ordered provider targets in open-sse/services/combo.ts.
• Streaming responses include keep-alive frames via withEarlyStreamKeepalive to maintain SSE connections during long generations.

Frequently Asked Questions

How does OmniRoute handle streaming versus non-streaming requests?

The pipeline checks for stream: true in the request body or an Accept: text/event-stream header at the API route level. For non-streaming requests, the executor returns the complete translated JSON; for streaming requests, the response passes through withEarlyStreamKeepalive (src/app/api/v1/chat/completions/route.ts) to emit periodic ping frames and prevent gateway timeouts.

What security measures are applied before a request reaches the upstream provider?

OmniRoute applies three layers of defense: admission control (admitChatRequest) enforces payload size limits and capacity reservations; prompt-injection detection (src/middleware/promptInjectionGuard.ts) scans for malicious patterns; and content-type validation rejects non-JSON bodies with HTTP 415 before parsing occurs.

How does the combo routing system work for provider fallback?

The combo resolver in open-sse/services/combo.ts expands a combo ID into an ordered list of concrete targets (provider + model + account). The handleChatCore function iterates through these targets sequentially; if one provider fails or times out, the pipeline automatically retries with the next target in the list until success or exhaustion.

What happens if the upstream provider returns an error?

The DefaultExecutor (open-sse/executors/default.ts) applies exponential backoff retry logic for transient failures. For non-retryable errors (HTTP 4xx or final retry exhaustion), the executor surfaces the error as a standardized OmniRoute error object with the upstream status code and message, which the API route returns to the client without exposing internal stack traces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}