OmniRoute Request Flow: How Open-SSE Handlers Route Requests to Executors

OmniRoute processes chat completion requests through a multi-stage pipeline in the open-sse workspace, starting from the Next.js API route, passing through admission guards and transformation layers, before handing off to provider-specific executors that handle upstream LLM calls.

The open-sse package in OmniRoute implements a modular architecture where each stage of request processing is isolated into small, testable functions. Understanding this flow is essential for developers extending the proxy, debugging provider-specific issues, or implementing custom middleware. This guide traces the exact path from HTTP request to upstream executor invocation using the release/v3.8.50 branch.

Overview of the Request Pipeline

A POST to /v1/chat/completions (or /v1/responses) traverses three architectural layers:

1. Public Route Layer – CORS, admission, and initial validation
2. Handler Layer – Core orchestration via handleChatCore
3. Executor Layer – Provider-specific upstream communication

Each layer delegates to the next through explicit function calls, with no implicit middleware magic.

Layer 1: Next.js Public Route and Admission

The entry point is src/app/api/v1/chat/completions/route.ts (lines 71-99). This thin route handler performs protocol-level concerns before delegating to business logic:

• CORS preflight handling for cross-origin requests
• Content-Type validation ensuring proper JSON bodies
• Heavy-weight admission checks for capacity limits and request size
• Prompt-injection guard – first line of defense against jailbreaking attempts
• Streaming detection from the stream parameter to determine response handling strategy

Once validated, the route calls handleChat from @/sse/handlers/chat, which immediately forwards to handleChatCore.

// From src/app/api/v1/chat/completions/route.ts (simplified)
import { handleChat } from "@/sse/handlers/chat";

export async function POST(request: Request) {
  // CORS, validation, admission checks...
  const response = await handleChat({ body, modelInfo, credentials, log });
  return new Response(response.body, { headers: response.headers });
}

Layer 2: Core Handler Orchestration in chatCore.ts

open-sse/handlers/chatCore.ts contains handleChatCore, the central orchestrator that sequences 13+ discrete processing steps. Each sub-step resides in its own file for isolated testing.

Step 2a: Request Setup and Routing Metadata

requestSetup.ts extracts routing decisions from the incoming request:

• Target provider (openai, anthropic, gemini, etc.)
• Model identifier and combo routing rules
• Extended context flags for special handling modes

Step 2b: Format Resolution

Two complementary modules determine transformation requirements:

• requestFormat.ts – Infers the source format from request shape (OpenAI-compatible, Anthropic Messages, Gemini, etc.)
• targetFormat.ts – Determines the target format the selected provider expects

This dual-resolution enables OmniRoute to translate between arbitrary provider schemas.

Step 2c: Early Cache Short-Circuits

Before expensive processing, two cache layers are consulted:

• Idempotency cache (idempotency.ts) – Returns identical responses for replayed requests with the same idempotency key
• Semantic cache (semanticCache.ts) – Vector-similarity matching for semantically equivalent prompts

Either can return a cached response and terminate the pipeline early.

Step 2d: Guardrails and Plugin Hooks

Custom policy enforcement runs via:

• runPluginOnRequestHook – Transform or block requests based on configured plugins
• runPluginOnResponseHook – Scheduled for post-processing (invoked after upstream response)

Step 2e: Memory and Skills Injection

memorySkillsInjection.ts augments the request body with:

• Persistent memory from prior conversations
• Skill call definitions for A2A (agent-to-agent) capabilities

Step 2f: Proactive Prompt Compression

When prompts exceed ~70% of the model's token limit, compression/strategySelector.ts executes a tiered pipeline:

Stage	Strategy	Intensity
1	Lite	Minimal semantic preservation
2	Standard	Balanced compression
3	Caveman	Aggressive simplification
4	Aggressive	Maximum token reduction

The body is replaced with the compressed version if any stage succeeds.

Step 2g-h: Translation and Upstream Body Assembly

• translateRequest (from translator/index.ts) converts the unified internal schema to provider-specific JSON
• upstreamBody.ts assembles the exact payload for the HTTP request to the LLM provider

Step 2i: Header Construction

upstreamExecuteHeaders.ts builds per-request headers including:

• Authentication tokens (Authorization, x-api-key, etc.)
• Custom User-Agent strings
• Cache-control overrides
• Provider-specific metadata headers

Step 2j-k: Executor Selection and Provider Call

// From open-sse/executors/index.ts
import { getExecutor } from "@/sse/executors";

const executor = getExecutor(providerId);  // "openai" → DefaultExecutor
const upstreamResponse = await executor.execute({
  url: providerEndpoint,
  headers: upstreamHeaders,
  body: upstreamBody,
});

The getExecutor factory (in open-sse/executors/index.ts) selects the appropriate class:

• DefaultExecutor – OpenAI-compatible providers (most common)
• CursorExecutor – Cursor-specific handling
• CodexExecutor – OpenAI Codex endpoints
• VertexExecutor – Google Cloud Vertex AI

All executors inherit from BaseExecutor (open-sse/executors/base.ts), which implements:

• URL construction with path templates
• Request transformer application
• Fetch with retry/back-off
• Timeout enforcement
• Circuit-breaker logic for failing providers

Step 2l: Response Handling Paths

Streaming responses (stream: true) flow through streamingPipeline.ts:

// Simplified streaming pipeline assembly
const transformStream = createSSETransformStreamWithLogger(log);
const usageStream = streamingUsageStats(modelInfo);
const costStream = streamingCost(pricing);
// TransformStream chain: raw bytes → parsed SSE → usage accounting → cost tracking → client

• createSSETransformStreamWithLogger – Parses raw bytes to SSE events with logging
• createPassthroughStreamWithLogger – Optional debugging passthrough
• Heartbeat injection for connection keepalive
• streamingResponseHeaders.ts – Final HTTP header assembly

Non-streaming responses use:

• nonStreamingResponseParse.ts – Full body consumption
• nonStreamingResponseHeaders.ts – Header construction
• Synchronous usage and cost recording

Step 2m: Post-Processing and Analytics

Before returning to the client:

• Semantic cache storage for future similar requests
• Compressed usage receipt attachment in response metadata
• Quota-share updates for multi-tenant accounting
• Gamification event emission for engagement tracking

Layer 3: Responses API Wrapper

The /v1/responses endpoint (OpenAI's newer API shape) uses responsesHandler.ts as an adapter:

1. Convert incoming Responses API format to Chat Completions format
2. Call handleChatCore (steps 2a-2m above)
3. Transform the upstream SSE stream back via createResponsesApiTransformStream

This avoids code duplication while supporting both API specifications.

Complete Request Flow Visualization

POST /v1/chat/completions
        │
        ▼
┌─────────────────┐
│  route.ts       │ CORS, admission, prompt-injection guard
│  (lines 71-99)  │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  handleChat     │ Thin wrapper
│  → handleChatCore│
└────────┬────────┘
         │
    ┌────┴────┬────────┬────────┬────────┐
    ▼         ▼        ▼        ▼        ▼
requestSetup  format   caches   plugins  memory/
              resolution        guardrails skills
    └────┬────┴────────┴────────┴────────┘
         ▼
    compression/strategySelector.ts (if needed)
         │
         ▼
    translator/index.ts ──► upstreamBody.ts ──► upstreamExecuteHeaders.ts
         │
         ▼
    executors/index.ts → getExecutor(provider) → executor.execute()
                                                    │
                                                    ▼
                                              LLM Provider API
                                                    │
         ┌──────────────────────────────────────────┘
         ▼
    streamingPipeline.ts OR non-streaming handlers
         │
         ▼
    semantic cache store, quota update, analytics
         │
         ▼
    Response to client

Code Examples

Minimal cURL to Exercise Full Pipeline

curl -X POST https://omniroute.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OMNI_KEY" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role": "user", "content": "Explain request routing"}],
    "stream": true
  }'

Direct Core Handler Invocation (Testing)

import { handleChatCore } from "@/sse/handlers/chatCore";

const response = await handleChatCore({
  body: {
    model: "claude-3-opus-20240229",
    messages: [{ role: "user", content: "Hello" }],
    stream: false,
  },
  modelInfo: { provider: "anthropic", model: "claude-3-opus-20240229" },
  credentials: { apiKey: process.env.ANTHROPIC_API_KEY },
  log: console,
});
// response is a native Response object ready to return to client

Executor Factory Usage

import { getExecutor } from "@/sse/executors";

async function callProvider(body: unknown, provider: string, url: string) {
  const executor = getExecutor(provider);  // DefaultExecutor, VertexExecutor, etc.
  
  return executor.execute({
    url,
    headers: { Authorization: `Bearer ${getApiKey(provider)}` },
    body,
    timeoutMs: 60000,
  });
}

Key Source Files Reference

File	Responsibility	Branch Link
src/app/api/v1/chat/completions/route.ts	Public API surface, admission	route.ts
open-sse/handlers/chatCore.ts	Central orchestration logic	chatCore.ts
open-sse/handlers/chatCore/requestSetup.ts	Routing metadata extraction	requestSetup.ts
open-sse/handlers/chatCore/idempotency.ts	Request deduplication	idempotency.ts
open-sse/services/compression/strategySelector.ts	Prompt compression pipeline	strategySelector.ts
open-sse/translator/index.ts	Cross-provider schema translation	translator/index.ts
open-sse/executors/index.ts	Executor factory	executors/index.ts
open-sse/executors/base.ts	Base executor with retry/circuit logic	base.ts
open-sse/handlers/chatCore/streamingPipeline.ts	SSE stream transformation	streamingPipeline.ts

Summary

The OmniRoute open-SSE request flow implements a deliberately granular pipeline:

• Admission and guards run at the edge before expensive processing
• Cache layers can short-circuit 80%+ of identical requests
• Compression and translation adapt arbitrary prompts to provider constraints
• Executor abstraction unifies retry, timeout, and circuit-breaker semantics across dozens of LLM providers
• Streaming and non-streaming paths share core logic but apply appropriate response handling

This architecture enables fine-grained testing (each file has unit tests), runtime observability (every stage emits structured logs), and safe extensibility (new providers add an executor class; new policies inject at plugin hooks).

Frequently Asked Questions

How does OmniRoute handle provider-specific API differences?

OmniRoute uses two-phase translation: requestFormat.ts and targetFormat.ts identify source and target schemas, then translator/index.ts applies the transformation. Provider-specific executors (open-sse/executors/) handle URL patterns, authentication schemes, and response parsing that can't be generalized.

What happens when a request exceeds token limits?

The compression/strategySelector.ts module proactively compresses prompts exceeding ~70% of the model's context window. It attempts four strategies in ascending intensity (lite → standard → caveman → aggressive), stopping at the first that brings the prompt under threshold. Each strategy is a pure function tested in isolation.

Can the pipeline be extended with custom middleware?

Yes. The plugin system exposes runPluginOnRequestHook and runPluginOnResponseHook injection points in open-sse/handlers/chatCore/. Plugins receive the full request context and can modify, log, or reject requests. Custom executors can be registered in executors/index.ts for provider-specific behavior.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}