Understanding the Request Pipeline Architecture in OmniRoute: From HTTP Request to Streamed Response

OmniRoute’s request pipeline architecture processes incoming chat completion requests through 11 distinct stages, transforming HTTP calls into provider-agnostic streaming responses while enforcing security, capacity limits, and usage tracking.

OmniRoute is an open-source AI gateway that normalizes requests across multiple LLM providers. Its request pipeline architecture handles everything from CORS validation to token usage persistence, ensuring reliable delivery of chat completions through a modular, middleware-based design.

The 11-Stage Request Processing Flow

OmniRoute’s pipeline handles a POST request to /v1/chat/completions through sequential stages, each isolated in specific modules for maintainability and observability.

1. HTTP Entry and CORS Validation

The pipeline begins at src/app/api/v1/chat/completions/route.ts, where the route handler validates Content-Type headers and handles CORS pre-flight requests. Non-JSON bodies receive an immediate 415 response, while valid requests proceed to admission control.

2. Admission and Capacity Throttling

Before processing, src/shared/middleware/chatBodyAdmission.ts enforces capacity limits via the admitChatRequest function. This stage builds a request-lease that auto-releases when the request completes, protecting against DoS attacks through early-exit overload protection.

3. Prompt-Injection Guard

Security scanning occurs in src/middleware/promptInjectionGuard.ts through createInjectionGuard. Malicious payloads trigger an immediate 400 response before reaching downstream providers.

4. Model-Alias Resolution

User-friendly model names (e.g., gpt-4o) normalize to provider-specific identifiers in src/lib/modelAliasResolver.ts. The resolveModelAliasWithSeedFallbackOnBody function maintains a seeded list of 30+ aliases for automatic fallback when unknown models are requested.

5. Streaming Decision

The pipeline determines response format in src/shared/utils/aiSdkCompat.ts. The acceptHeaderForcesStream utility checks for stream:true or SSE Accept headers, preparing the early-keep-alive wrapper from src/open-sse/utils/earlyStreamKeepalive.ts for long-running connections.

6. Core Chat Handler

The handleChat function in src/sse/handlers/chat.ts orchestrates combo logic, provider credential selection, and fallback loops. This stage dispatches prepared requests to the SSE core while handling error boundaries.

7. SSE Core Orchestration

open-sse/handlers/chatCore.ts contains the handleChatCore function, which translates requests to provider-specific formats using translators in open-sse/translator/*. This provider-agnostic layer enables seamless switching between OpenAI, Claude, Gemini, and other backends.

8. Provider Executor

Provider-specific executors in open-sse/executors/* (e.g., DefaultExecutor, AntigravityExecutor) extend the BaseExecutor pattern defined in open-sse/executors/base.ts. These handle HTTP dispatch, retries, token refresh, and circuit-breaker logic while respecting per-provider rate limits.

9. Stream Transformation

Response normalization occurs in open-sse/utils/stream.ts and streamHandler.ts, converting upstream SSE/JSON into OpenAI-compatible formats. The open-sse/handlers/responseSanitizer.ts removes unsafe fields and injects usage metadata.

10. Usage Extraction and Persistence

Token counting and cost tracking happen in open-sse/utils/usageTracking.ts, persisting records through src/lib/usageDb.ts. This captures provider, model, prompt tokens, and completion timestamps for audit logs and dashboards.

11. Response Wrapping

Final HTTP response construction adds correlation IDs and compression headers via src/shared/utils/compressionHeaderEcho.ts. For streaming requests, the keep-alive wrapper ensures connection stability throughout the lifecycle.

Key Implementation Details

Entry Point and Request Admission

The route handler in src/app/api/v1/chat/completions/route.ts wires the pipeline together:

import { handleChat } from "@/sse/handlers/chat";
import { generateRequestId } from "@/shared/utils/requestId";
import { admitChatRequest } from "@/shared/middleware/chatBodyAdmission";
import { acceptHeaderForcesStream } from "@/shared/utils/aiSdkCompat";
import { withEarlyStreamKeepalive } from "@/open-sse/utils/earlyStreamKeepalive";

export async function POST(request: Request) {
  await ensureInitialized();
  const admission = await admitChatRequest(request, { 
    sessionId: resolveSessionId(request) 
  });
  const parsedBody = await request.json();
  
  const wantsStreaming = parsedBody?.stream === true || 
    acceptHeaderForcesStream(request.headers.get("accept"));
  const reqId = generateRequestId();

  if (wantsStreaming) {
    const handler = handleChat(request, null, parsedBody, reqId);
    return withEarlyStreamKeepalive(handler);
  }

  return handleChat(request, null, parsedBody);
}

Provider Translation and Execution

The SSE core handles provider abstraction through translation layers:

// open-sse/handlers/chatCore.ts
export async function handleChatCore(
  body: ChatRequest,
  modelInfo: ResolvedModel,
  credentials: ProviderCredentials,
  requestId: string
) {
  const translated = await translateRequest(body, modelInfo);
  const execResult = await getExecutor(modelInfo.provider)
    .execute(translated, credentials);
  const stream = await translateResponse(execResult, requestId);
  return stream;
}

Usage Tracking Implementation

After provider response, usage extraction enables cost management:

// open-sse/utils/usageTracking.ts
export async function extractUsage(
  stream: ReadableStream, 
  meta: { provider: string; model: string }
) {
  const usage = await parseUsageFromStream(stream);
  await usageDb.record({
    provider: meta.provider,
    model: meta.model,
    promptTokens: usage.promptTokens,
    completionTokens: usage.completionTokens,
    timestamp: Date.now(),
  });
}

Architecture Benefits

OmniRoute’s request pipeline architecture provides several operational advantages:

• Provider Agnosticism: The translation layer in open-sse/translator/* decouples client requests from provider-specific formats, enabling hot-swapping between LLM backends without client changes.
• Resilience Patterns: Base executors implement circuit-breakers, exponential backoff, and automatic token refresh, isolating failures to specific providers while maintaining service availability.
• Security-First Design: Prompt-injection scanning and response sanitization occur at pipeline boundaries, ensuring malicious content never reaches upstream providers or downstream clients.
• Observability: Correlation IDs flow through all 11 stages, with usage persistence enabling per-request cost attribution and capacity planning.

Summary

• OmniRoute processes chat completions through 11 distinct pipeline stages, from HTTP entry in route.ts to usage persistence in usageDb.ts.
• Capacity protection via admitChatRequest prevents overload, while prompt-injection guards block malicious payloads before provider dispatch.
• Model-alias resolution normalizes user-friendly names to provider IDs, supporting 30+ pre-seeded aliases with automatic fallback.
• Provider executors extend a BaseExecutor pattern that standardizes retries, authentication refresh, and rate-limit compliance across OpenAI, Claude, Gemini, and custom backends.
• Stream transformations guarantee OpenAI SDK compatibility regardless of upstream provider format, with keep-alive frames maintaining long-running connections.
• Usage tracking captures token counts and metadata for every request, enabling cost attribution and audit trails through src/lib/usageDb.ts.

Frequently Asked Questions

How does OmniRoute handle high-traffic load and prevent server overload?

OmniRoute implements admission control in src/shared/middleware/chatBodyAdmission.ts through the admitChatRequest function. This middleware creates a request-lease for each incoming connection, enforcing per-session limits and global capacity thresholds. When capacity is exceeded, the pipeline returns early with an overload response, protecting downstream providers and maintaining service stability for admitted requests.

What security measures exist in the request pipeline to prevent prompt injection attacks?

The pipeline includes a dedicated prompt-injection guard stage implemented in src/middleware/promptInjectionGuard.ts. Before any downstream processing occurs, the createInjectionGuard function scans the JSON payload for malicious patterns. If injection attempts are detected, the pipeline terminates immediately with a 400 response, ensuring harmful content never reaches LLM providers or training data.

Can OmniRoute translate between different LLM provider formats automatically?

Yes. The SSE Core Orchestration stage in open-sse/handlers/chatCore.ts uses provider-specific translators located in open-sse/translator/* to convert requests and responses bidirectionally. This means clients can use standard OpenAI SDK formats while the gateway automatically translates to Claude, Gemini, or other provider schemas. The translateRequest and translateResponse functions handle format normalization, enabling seamless provider switching without client code changes.

Where does usage tracking occur in the pipeline, and what data is captured?

Usage extraction happens in stage 10 through open-sse/utils/usageTracking.ts after the provider response is received but before final client delivery. The extractUsage function captures provider name, model identifier, prompt tokens, completion tokens, and timestamps. This data persists to src/lib/usageDb.ts, enabling real-time cost tracking, usage dashboards, and comprehensive audit logs for every chat completion request.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}