How to Structure a Large-Scale Application with OmniRoute: A Layered Monorepo Guide

OmniRoute is designed as a modular, layered monorepo that cleanly separates API routes, streaming logic, domain policies, persistence, and auxiliary services, enabling teams to build an extensible AI gateway that scales across dozens of providers.

OmniRoute (diegosouzapw/OmniRoute) is an open-source AI gateway built to expose an OpenAI-compatible API while orchestrating multiple upstream providers. Learning how to structure a large-scale application with OmniRoute means adopting its directory conventions and request lifecycle patterns that keep the public API, business rules, and provider integrations testable and decoupled.

High-Level Directory Layout

OmniRoute organizes code into ten distinct layers. Each layer owns a single responsibility and maps directly to the request lifecycle described in docs/architecture/ARCHITECTURE.md.

API Routes Layer

The API Routes layer lives in src/app/api/v1/ and exposes OpenAI-compatible endpoints such as /v1/chat/completions and /v1/embeddings, alongside management APIs like /api/providers and /api/combos. The entry point for chat completions is src/app/api/v1/chat/completions/route.ts, which validates incoming requests with Zod before delegating to the core handler.

SSE and Routing Core

The SSE / Routing Core under open-sse/handlers/ and open-sse/executors/ manages request parsing, model resolution, translation, and streaming to the client. The central orchestrator is open-sse/handlers/chatCore.ts, which dispatches provider-specific HTTP calls and manages response streams.

Domain and Policy Layer

The Domain / Policy layer in src/domain/ enforces centralized business rules before any provider call. Key modules include src/domain/comboResolver.ts for combo resolution and src/domain/policyEngine.ts for lockout, budgeting, and fallback checks.

Persistence Layer

The Persistence layer in src/lib/db/ uses a SQLite-backed state with a write-through cache pattern. All mutable state is stored in a single SQLite file (${DATA_DIR}/storage.sqlite). The connection, WAL setup, and migrations are handled by src/lib/db/core.ts, while domain-specific CRUD operations live in src/lib/db/domainState.ts and usage history is tracked via src/lib/usageDb.ts.

Provider Executors and Translators

The Provider Executors in open-sse/executors/ contain one executor per provider family, such as DefaultExecutor, AntigravityExecutor, and AzureOpenAIExecutor. Each implements URL building, header management, retry logic, and token refresh. The Translators under open-sse/translator/ convert provider-specific schemas to the canonical OpenAI format; for example, open-sse/translator/request/claude-to-openai.ts handles Claude request normalization.

Auth and Security

The Auth & Security layer in src/server/authz/ manages request classification, API-key enforcement, IP filtering, and route guarding. The main pipeline is defined in src/server/authz/pipeline.ts.

Auxiliary Services

Beyond the core gateway, OmniRoute supports three auxiliary service categories:

• Embedded Services (src/lib/services/): Optional locally-run AI tools like 9Router and CLIProxyAPI that appear as regular providers. The lifecycle manager is src/lib/services/ServiceSupervisor.ts.
• Cloud Agents (src/lib/cloudAgent/): DB-backed task lifecycle for remote agents such as Codex Cloud, Devin, and Jules, coordinated through src/lib/cloudAgent/registry.ts.
• Webhooks and Evaluation (src/lib/webhookDispatcher.ts, src/lib/evals/): Outbound event dispatch and automated quality-assurance suites.

Dashboard UI

The Dashboard UI in src/app/(dashboard)/dashboard/ provides React-based pages for provider management, combo editing, analytics, and settings. An example page is src/app/(dashboard)/dashboard/providers/page.tsx.

Request Lifecycle for Chat Completions

Understanding the request flow is essential when structuring a large-scale application with OmniRoute. A typical chat completion follows this path:

1. The client posts to /v1/chat/completions, handled by src/app/api/v1/chat/completions/route.ts.
2. Zod validation guarantees type safety before any business logic executes.
3. The route delegates to open-sse/handlers/chatCore.ts, which resolves the model and invokes the domain policy engine (src/domain/policyEngine.ts).
4. The policy engine enforces lockout, budget, and fallback checks before the provider call.
5. chatCore.ts invokes the appropriate provider executor (e.g., open-sse/executors/defaultExecutor.ts), which performs the upstream HTTP request.
6. Circuit breaker logic (src/shared/utils/circuitBreaker.ts) and account fallback logic (open-sse/services/accountFallback.ts) prevent cascading failures by retrying alternate credentials or models on transient errors.
7. The executor streams the response through the translator registry (open-sse/translator/) and generic stream helpers (open-sse/utils/stream.ts), normalizing chunks into OpenAI-compatible SSE format.
8. Token usage is extracted and persisted via src/lib/usageDb.ts.

Error sanitisation is enforced by open-sse/utils/error.ts, ensuring raw stack traces never leak to clients. Outbound requests are guarded by src/shared/network/safeOutboundFetch.ts and src/shared/network/outboundUrlGuard.ts to block SSRF-prone private URLs.

Combo Routing and Auto-Combo

OmniRoute supports named combos (static sequences) and Auto-Combo (dynamic, score-based selection). The engine lives under open-sse/services/autoCombo/ and evaluates 19 strategies using a nine-factor scoring model that weighs cost, latency, success rate, quota headroom, and breaker state.

• The engine entry point is open-sse/services/autoCombo/autoComboEngine.ts.
• Scoring logic resides in open-sse/services/autoCombo/scoringEngine.ts.
• The virtual factory inside autoComboEngine.ts materializes an ad-hoc combo when no static definition matches.

You can define a custom combo via a dedicated route:

// src/app/api/combos/my-cheap-combo/route.ts
import { defineCombo } from '@/open-sse/services/combo/defineCombo';

export const GET = async () => {
  await defineCombo({
    name: 'my-cheap-combo',
    models: [
      'openai/gpt-3.5-turbo',
      'anthropic/claude-instant-1.2',
      'xai/grok-7b',
    ],
    strategy: 'weighted',
    weights: [0.5, 0.3, 0.2],
  });
  return new Response('Combo created', { status: 201 });
};

Combo definitions are persisted in the combos table via src/lib/db/combo.ts.

To invoke Auto-Combo from a client, prefix the model name with auto/, such as auto/fast:

curl -X POST http://localhost:20128/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "auto/fast",
    "messages": [{ "role": "user", "content": "Write a short poem about clouds." }]
  }'

The auto/fast prefix triggers the Auto-Combo engine, which selects the cheapest low-latency models that satisfy the request. For scoring formulas and strategy details, see docs/routing/AUTO-COMBO.md.

Persistence Strategy

All mutable state is persisted in a single SQLite file (${DATA_DIR}/storage.sqlite). The DB layer follows a write-through cache pattern: in-memory Maps hold the current state, and every mutation is synchronously written to SQLite via the src/lib/db/* modules. This guarantees crash-safety while keeping runtime reads fast.

Key modules include:

• src/lib/db/core.ts – connection, WAL setup, and migrations.
• src/lib/db/domainState.ts – CRUD for domain tables such as fallback chains, budgets, and lockouts.
• src/lib/usageDb.ts – usage-history facade for the usage_history table.

Extending the System

When adding a new provider, combo strategy, or UI tab, follow OmniRoute's layered convention:

1. Register the provider in open-sse/config/providerRegistry.ts or the OAuth registry under src/lib/oauth/providers/.
2. Create an executor that extends BaseExecutor in a new file such as open-sse/executors/myProviderExecutor.ts.
3. Add request/response translators if the provider's schema differs from OpenAI.
4. Expose a route under src/app/api/v1/providers/[myProvider]/chat/completions/route.ts if a dedicated per-provider endpoint is required.
5. Write a unit test in tests/unit/ that exercises the new flow; the test suite verifies lint, type-check, and coverage gates automatically.

For example, adding a new OAuth provider such as "MyAI" requires a definition file and an index re-export:

// src/lib/oauth/providers/myai.ts
import { OAuthProvider } from '@/src/lib/oauth/types';

export const myai: OAuthProvider = {
  name: 'myai',
  authUrl: 'https://login.myai.com/auth',
  tokenUrl: 'https://login.myai.com/token',
  clientIdEnv: 'MYAI_CLIENT_ID',
  clientSecretEnv: 'MYAI_CLIENT_SECRET',
  scopes: ['api.read', 'api.write'],
};

After re-exporting in src/lib/oauth/providers/index.ts, run npm run typecheck:core to ensure the new type flows through the system.

Security and Resilience

OmniRoute implements a three-layer resilience model spanning provider, connection, and model scopes, as detailed in docs/security/RESILIENCE_GUIDE.md.

• Circuit breakers (src/shared/utils/circuitBreaker.ts) prevent flapping providers from overwhelming the system.
• Connection cooldowns (open-sse/services/accountFallback.ts) handle transient errors such as 429 or 500 by retrying alternate credentials.
• Error sanitisation (open-sse/utils/error.ts) strips raw stack traces from client-facing responses.
• Outbound fetch guards (src/shared/network/safeOutboundFetch.ts and src/shared/network/outboundUrlGuard.ts) block SSRF-prone private URLs.

Deployment Topology

OmniRoute runs as a single Node.js process combining Next.js and the SSE core. It can be containerized, run on a developer workstation, or deployed on a VM. The architecture follows a three-tier interaction between Clients → OmniRoute → Upstream Providers, with optional Cloud Sync and Embedded Services attached as sidecars.

To start a minimal OmniRoute server:

import { createServer } from 'http';
import next from 'next';

const dev = process.env.NODE_ENV !== 'production';
const app = next({ dev, hostname: '0.0.0.0', port: 20128 });
const handle = app.getRequestHandler();

await app.prepare();

createServer((req, res) => {
  // All Next.js API routes (including /v1/*) are handled automatically
  handle(req, res);
}).listen(20128, () => {
  console.log('🚀 OmniRoute listening on http://localhost:20128');
});

This pattern is implemented in src/server.ts.

Summary

• OmniRoute uses a layered monorepo structure that isolates API routes (src/app/api/v1/), routing core (open-sse/handlers/), domain policies (src/domain/), and persistence (src/lib/db/).
• The request lifecycle enforces Zod validation, domain policy checks, circuit breakers, and account fallback before streaming normalized responses back to the client.
• Combo routing supports both static named combos and dynamic Auto-Combo selection via open-sse/services/autoCombo/autoComboEngine.ts.
• SQLite persistence follows a write-through cache pattern for crash-safe, high-performance state management.
• Extending the gateway follows a strict convention: register the provider, create an executor, add translators, expose routes, and write tests.

Frequently Asked Questions

What is the best way to add a new AI provider to OmniRoute?

Register the provider in open-sse/config/providerRegistry.ts (or the OAuth registry under src/lib/oauth/providers/), then create an executor extending BaseExecutor in open-sse/executors/. Add request and response translators if the schema deviates from OpenAI, expose a dedicated route under src/app/api/v1/providers/ if needed, and add unit tests in tests/unit/.

How does OmniRoute handle provider failures and retries?

OmniRoute applies a three-scope resilience model using circuit breakers (src/shared/utils/circuitBreaker.ts) at the provider level and account fallback logic (open-sse/services/accountFallback.ts) at the connection level. If an upstream returns a transient error such as 429 or 500, the system automatically retries an alternate credential or model before returning an error to the client.

Where does OmniRoute store configuration and usage data?

All mutable state lives in a single SQLite database (${DATA_DIR}/storage.sqlite) managed by src/lib/db/core.ts. The architecture uses a write-through cache so that in-memory Maps serve fast reads while SQLite guarantees durability for provider connections, model aliases, combos, usage history, and circuit-breaker state.

Can OmniRoute dynamically select models instead of using static routing?

Yes. The Auto-Combo engine in open-sse/services/autoCombo/autoComboEngine.ts dynamically evaluates 19 strategies across a nine-factor scoring model. By sending a request with a model prefix such as auto/fast, the engine selects the optimal provider based on real-time cost, latency, success rate, quota headroom, and breaker state.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}