OmniRoute Request Pipeline Architecture: 11-Stage Flow from HTTP to Streamed Response

OmniRoute's request pipeline transforms incoming /v1/chat/completions calls into streaming or JSON responses through 11 sequential stages spanning authentication, policy enforcement, model resolution, provider translation, and usage tracking.

Built by diegosouzapw/OmniRoute, this pipeline delivers a provider-agnostic LLM gateway that normalizes requests across OpenAI, Claude, Gemini, and other providers while maintaining strict OpenAI SDK compatibility. The architecture emphasizes resilience, observability, and security at every layer.

---

Pipeline Overview: From HTTP Entry to Final Response

The complete flow executes in src/app/api/v1/chat/completions/route.ts and delegates through specialized modules. Each stage is designed to fail fast, preserve context, and release resources properly.

---

Stage 1: HTTP Entry and CORS Handling

The POST and OPTIONS handlers in src/app/api/v1/chat/completions/route.ts validate Content-Type headers and enforce CORS pre-flight checks. Non-JSON bodies immediately receive a 415 Unsupported Media Type response.

// Entry point from route.ts
export async function POST(request: Request) {
  await ensureInitialized(); // translators initialization
  const parsedBody = await request.json();
  // ... downstream processing
}

---

Stage 2: Admission and Capacity Throttling

admitChatRequest and admitChatStructure in src/shared/middleware/chatBodyAdmission.ts guarantee system capacity through request-leases. These leases are acquired before processing and released on completion, protecting against DoS attacks and overload scenarios.

• Early-exit on capacity exhaustion
• Per-session request limits
• Automatic lease cleanup via finally blocks

---

Stage 3: Prompt-Injection Guard

Before any downstream processing, createInjectionGuard in src/middleware/promptInjectionGuard.ts scans the JSON payload for malicious patterns. Blocked requests return 400 Bad Request with security context.

---

Stage 4: Model-Alias Resolution

resolveModelAliasWithSeedFallbackOnBody in src/lib/modelAliasResolver.ts converts user-friendly names like gpt-4o into provider-specific identifiers. The system maintains a seeded fallback list of 30+ aliases for unknown model names, ensuring graceful degradation.

Input Alias	Resolved Provider String
gpt-4o	openai/gpt-4o-2024-05-13
claude-3-opus	anthropic/claude-3-opus-20240229
gemini-pro	google/gemini-1.0-pro

---

Stage 5: Streaming Decision

The pipeline detects streaming intent through:

• parsedBody.stream === true in request body
• acceptHeaderForcesStream in src/shared/utils/aiSdkCompat.ts checking the Accept header

When streaming is requested, withEarlyStreamKeepalive from src/open-sse/utils/earlyStreamKeepalive.ts wraps the handler to inject keep-alive frames preventing timeouts on long-running generations.

const wantsStreaming = parsedBody?.stream === true 
  || acceptHeaderForcesStream(request.headers.get("accept"));

if (wantsStreaming) {
  const handler = handleChat(request, null, parsedBody, reqId);
  return withEarlyStreamKeepalive(handler, { maxDurationMs: 30000 });
}

---

Stage 6: Core Chat Handler Orchestration

handleChat in src/sse/handlers/chat.ts performs the heavy lifting:

• Parses and validates request body
• Resolves combo configurations (multi-provider fallbacks)
• Selects appropriate provider credentials
• Dispatches to the SSE core with error handling loops

This stage handles fallback logic when primary providers fail, cycling through alternatives automatically.

---

Stage 7: SSE Core and Provider Translation

handleChatCore in open-sse/handlers/chatCore.ts operates as the provider-agnostic translation layer:

export async function handleChatCore(
  body: ChatRequest,
  modelInfo: ResolvedModel,
  credentials: ProviderCredentials,
  requestId: string
) {
  const translated = await translateRequest(body, modelInfo);
  const execResult = await getExecutor(modelInfo.provider).execute(translated, credentials);
  const stream = await translateResponse(execResult, requestId);
  return stream;
}

The open-sse/translator/ directory contains request/response converters for each supported provider, enabling seamless format translation between OpenAI, Claude, Gemini, and others.

---

Stage 8: Provider Execution with Resilience

Each provider implements an executor in open-sse/executors/:

Executor	Provider	Specialization
DefaultExecutor	Generic OpenAI-compatible	Standard retry and back-off
AntigravityExecutor	Antigravity AI	Custom token refresh handling
BaseExecutor	All (abstract)	Circuit-breaker, rate-limit respect, cooldown

The BaseExecutor pattern centralizes:

• Exponential backoff with jitter
• OAuth token refresh before expiry
• Circuit-breaker state management
• Per-provider rate limit tracking

---

Stage 9: Stream Transformation and Sanitization

open-sse/utils/stream.ts and streamHandler.ts normalize upstream SSE streams into strict OpenAI-compatible format. The responseSanitizer removes unsafe fields and injects proper SSE framing.

Key transformations:

• Provider-specific event formats → OpenAI SSE
• Usage metadata injection
• Response field filtering per security policies

---

Stage 10: Usage Extraction and Persistence

Post-execution, extractUsage in open-sse/utils/usageTracking.ts parses token counts from the stream. src/lib/usageDb.ts persists records for cost tracking dashboards and audit logs:

export async function extractUsage(stream: ReadableStream, meta: { provider: string; model: string }) {
  const usage = await parseUsageFromStream(stream);
  await usageDb.record({
    provider: meta.provider,
    model: meta.model,
    promptTokens: usage.promptTokens,
    completionTokens: usage.completionTokens,
    timestamp: Date.now(),
  });
}

---

Stage 11: Response Wrapping and Delivery

Final stage responsibilities:

• Correlation ID attachment (generateRequestId)
• Compression header echo via src/shared/utils/compressionHeaderEcho.ts
• Keep-alive wrapper termination for clean stream closure
• Proper Content-Type and caching headers

---

Key Files and Their Responsibilities

File Path	Pipeline Role
src/app/api/v1/chat/completions/route.ts	HTTP entry, CORS, admission orchestration
src/shared/middleware/chatBodyAdmission.ts	Capacity throttling and lease management
src/middleware/promptInjectionGuard.ts	Security filtering
src/lib/modelAliasResolver.ts	Model name normalization
src/sse/handlers/chat.ts	Combo handling and fallback orchestration
open-sse/handlers/chatCore.ts	Translation and executor dispatch
open-sse/translator/index.ts	Provider converter registry
open-sse/executors/base.ts	Shared resilience logic
open-sse/executors/*.ts	Provider-specific execution
open-sse/utils/earlyStreamKeepalive.ts	Streaming timeout prevention
src/lib/usageDb.ts	Cost and audit persistence
docs/architecture/ARCHITECTURE.md	Architecture documentation and diagrams

---

Summary

• 11 sequential stages process every /v1/chat/completions request in OmniRoute
• Fail-fast security via prompt-injection guards and capacity throttling
• Provider abstraction through translation layers and executor patterns
• Resilience built-in: circuit-breakers, retries, fallbacks, and keep-alive mechanisms
• Full observability: correlation IDs, usage tracking, and audit logging
• OpenAI SDK compatibility maintained regardless of upstream provider

The pipeline's modular design allows individual stages to evolve independently while preserving the contract between HTTP entry and streamed response.

---

Frequently Asked Questions

What triggers streaming mode in OmniRoute?

Streaming activates when the request body contains "stream": true or when the Accept header indicates SSE preference (detected by acceptHeaderForcesStream in src/shared/utils/aiSdkCompat.ts). The withEarlyStreamKeepalive wrapper then ensures the connection remains alive during long generations.

How does OmniRoute handle provider failures?

The handleChat function in src/sse/handlers/chat.ts implements combo logic with automatic fallback loops. If the primary provider's executor throws, the pipeline cycles through configured alternatives. Each executor inherits resilience patterns from BaseExecutor including circuit-breakers and exponential backoff.

Where are model aliases defined and resolved?

Aliases resolve in src/lib/modelAliasResolver.ts via resolveModelAliasWithSeedFallbackOnBody. The system maps friendly names like gpt-4o to provider-specific identifiers, falling back to a seeded list of 30+ aliases when encountering unknown values. This enables provider-agnostic client requests.

How is token usage tracked for billing?

The extractUsage utility in open-sse/utils/usageTracking.ts parses token counts from provider responses. src/lib/usageDb.ts persists records with provider, model, prompt tokens, completion tokens, and timestamp. This data powers cost-tracking dashboards and satisfies audit requirements.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}