How to Debug Issues in the OmniRoute Service Layer: A Complete Guide

Enable LOG_LEVEL=debug and trace the combo engine's execution through open-sse/services/combo.ts to identify why specific providers are skipped or requests fail.

Debugging issues in the OmniRoute service layer requires navigating the combo engine that routes every API request through the open-sse/services package. The service layer transforms each call into a combo—a prioritized list of target models with a routing strategy—and executes it via the core logic in combo.ts. This guide shows you how to debug issues in the OmniRoute service layer using the source code from the diegosouzapw/OmniRoute repository.

Enable Verbose Logging with Pino

OmniRoute uses the pino logger for structured output. Set the environment variable before starting the server to expose internal routing decisions.

LOG_LEVEL=debug omniroute start

With debug logging enabled, you will see COMBO-prefixed entries that trace the execution flow. The logger streams all log.info, log.warn, and log.debug calls to stdout, including the resolution of target models and fallback triggers.

Trace the Combo Resolution Process

Every request begins with combo resolution in open-sse/services/combo.ts. The function resolveComboTargets (lines ~331-350) expands wildcards and produces the resolvedComboTargets list that determines which providers are eligible.

When you issue a request, look for the initial log entry indicating the combo name and strategy. The engine then applies the selected routing algorithm—such as weighted selection or sticky routing—to the target list.

Inspect Target Selection Logic

When a target is skipped, the engine calls isTargetSelectableForWeighted (lines ~445-560). This helper checks for circuit breaker states, provider cooldowns, and model lock-outs. Look for log lines containing:

• "provider in cooldown"
• "circuit breaker open"
• "model locked"

These entries indicate why a specific provider was rejected before the request was dispatched.

Verify Quota Preflight Decisions

The buildAutoCandidates function (lines ~2910-3000) fetches quota information via fetchResetAwareQuotaWithCache. It calls evaluateQuotaCutoff to filter candidates based on remaining quota.

If you suspect quota issues, set RESILIENCE_SETTINGS.quotaPreflight.enabled=true to force the cutoff logic. The logs will show "quota_exhausted" when a candidate is blocked due to insufficient quota.

Diagnose Fallback and Retry Behavior

When a target returns a non-2xx status, the combo engine enters the retry loop managed by dispatchWithCooldownRetry. Look for the log entry "All targets failed — retrying set" around lines ~1240-1246 to identify when the engine cycles to the next retry set.

Each retry attempt logs the current count against maxSetRetries and the retryDelayMs value from the combo configuration. If all targets fail across all retry sets, the engine returns an unavailableResponse or errorResponse (lines ~620-640).

Query Runtime Metrics and Health

Access Combo Metrics Database

After execution, recordComboRequest in open-sse/services/comboMetrics.ts persists data to the combo_metrics SQLite table. You can query this data directly or use the CLI:

omniroute combo metrics get <combo-name>

The metrics include latency, token counts, success rates, and failure reasons.

Run the Diagnostic CLI

Use the built-in health check to verify provider status and circuit breaker states:

omniroute doctor --output json

This command reports the status of rate limits, cooldown windows, and quota availability across all configured providers.

Validate Request Entry Points

If a request never reaches the combo engine, verify it passed the Zod validation schemas in src/app/api/v1/**/route.ts. These files validate the request body and authentication before calling the service layer. Malformed requests are rejected with validation errors before combo.ts processes them.

Code Examples

Enabling Debug Logging and Reproducing Requests

# Start with maximum verbosity

LOG_LEVEL=debug omniroute start &

# Issue a failing request

curl -X POST http://localhost:20128/v1/chat/completions \
     -H "Authorization: Bearer <TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'

Expected log output includes:

COMBO  ▶️  routing directly to pinned context model: auto
COMBO  ⚙️  Weighted selection (sticky): 12 total targets
COMBO  ⏳  All targets failed — retrying set (1/3)
COMBO  ✅  Successful response from provider: openai/gpt-4o

Inspecting Resolved Targets Programmatically

import { resolveComboTargets, getComboFromData } from "./open-sse/services/combo.ts";

const combo = await getComboFromData("my-auto-combo");
const allCombos = []; // Fetch all combos if nested resolution is needed
const targets = await resolveComboTargets(combo, allCombos, 10);

console.log("Resolved targets:", targets.map(t => t.modelStr));

This expands wildcards via expandProviderWildcardsInCombo and returns the final routing list.

Triggering Quota Preflight for Testing

import { buildAutoCandidates } from "./open-sse/services/combo.ts";

const candidates = await buildAutoCandidates(
  resolvedTargets,
  "my-auto-combo",
  "session-123",
  undefined, // default reset-window config
  null       // default resilience settings
);

candidates.forEach(c => {
  console.log(`${c.provider}/${c.model} – quota ${c.quotaRemaining}%`);
});

Key Files for Debugging

File	Purpose
open-sse/services/combo.ts	Core combo handling, routing strategies, and fallback logic
open-sse/services/comboMetrics.ts	Persists latency, token usage, and success/failure counts
open-sse/services/accountFallback.ts	Circuit breaker and provider cooldown helpers
open-sse/services/rateLimitSemaphore.ts	Global rate limiting state
open-sse/services/providerCooldownTracker.ts	Per-provider cooldown tracking
open-sse/services/quotaPreflight.ts	Hard quota cutoff implementation
src/lib/db/comboMetrics.ts	SQLite schema for metrics storage
src/lib/events/eventBus.ts	Central event emitter for debug events

Summary

• Enable LOG_LEVEL=debug to see the combo engine's routing decisions and skip reasons.
• Trace execution through open-sse/services/combo.ts, starting with resolveComboTargets (lines ~331-350).
• Check isTargetSelectableForWeighted (lines ~445-560) to identify why providers are skipped due to cooldowns, circuit breakers, or locks.
• Inspect quota logic in buildAutoCandidates (lines ~2910-3000) when candidates are filtered due to quota limits.
• Use omniroute doctor for a snapshot of provider health and circuit breaker states.
• Validate entry points in src/app/api/v1/**/route.ts if requests fail before reaching the service layer.

Frequently Asked Questions

Why is my request returning a 503 Service Unavailable error?

A 503 typically indicates that the combo engine exhausted all retry sets without finding a healthy provider. Check open-sse/services/combo.ts around lines ~1240-1260 for the "All targets failed" log entry. This occurs when all candidates are in cooldown, circuit breakers are open, or quotas are exhausted. Run omniroute doctor to verify provider health and circuit breaker states.

How do I see which provider was selected for my request?

Look for the COMBO log entries that show the resolved target list after resolveComboTargets executes (lines ~331-350). If using weighted selection, the logs will indicate the chosen provider after isTargetSelectableForWeighted filters the list. For sticky routing, the logs will show the pinned context model being reused.

Where are combo execution metrics stored?

Metrics are stored in a SQLite database via recordComboRequest in open-sse/services/comboMetrics.ts. You can query the combo_metrics table directly or use the CLI command omniroute combo metrics get <combo-name>. The data includes request latency, token usage, and success rates for each combo execution.

How can I check if a provider is in cooldown?

Provider cooldown state is tracked in open-sse/services/providerCooldownTracker.ts. When debugging, look for log entries containing "provider in cooldown" in the output from isTargetSelectableForWeighted (lines ~445-560). You can also check the global cooldown state via omniroute doctor, which reports cooldown windows and remaining seconds for each provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}