# How to Debug Issues in the OmniRoute Service Layer: A Complete Guide

> Debug OmniRoute service layer issues effectively. Set LOG_LEVEL debug and trace combo engine execution in combo.ts to pinpoint skipped providers or failed requests.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: how-to-guide
- Published: 2026-07-06

---

**Enable `LOG_LEVEL=debug` and trace the combo engine's execution through [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) to identify why specific providers are skipped or requests fail.**

Debugging issues in the OmniRoute service layer requires navigating the **combo engine** that routes every API request through the `open-sse/services` package. The service layer transforms each call into a combo—a prioritized list of target models with a routing strategy—and executes it via the core logic in [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts). This guide shows you how to debug issues in the OmniRoute service layer using the source code from the `diegosouzapw/OmniRoute` repository.

## Enable Verbose Logging with Pino

OmniRoute uses the **pino** logger for structured output. Set the environment variable before starting the server to expose internal routing decisions.

```bash
LOG_LEVEL=debug omniroute start

```

With debug logging enabled, you will see `COMBO`-prefixed entries that trace the execution flow. The logger streams all `log.info`, `log.warn`, and `log.debug` calls to stdout, including the resolution of target models and fallback triggers.

## Trace the Combo Resolution Process

Every request begins with **combo resolution** in [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts). The function `resolveComboTargets` (lines ~331-350) expands wildcards and produces the `resolvedComboTargets` list that determines which providers are eligible.

When you issue a request, look for the initial log entry indicating the combo name and strategy. The engine then applies the selected routing algorithm—such as weighted selection or sticky routing—to the target list.

### Inspect Target Selection Logic

When a target is skipped, the engine calls `isTargetSelectableForWeighted` (lines ~445-560). This helper checks for **circuit breaker** states, **provider cooldowns**, and **model lock-outs**. Look for log lines containing:

- `"provider in cooldown"`
- `"circuit breaker open"`
- `"model locked"`

These entries indicate why a specific provider was rejected before the request was dispatched.

### Verify Quota Preflight Decisions

The `buildAutoCandidates` function (lines ~2910-3000) fetches quota information via `fetchResetAwareQuotaWithCache`. It calls `evaluateQuotaCutoff` to filter candidates based on remaining quota.

If you suspect quota issues, set `RESILIENCE_SETTINGS.quotaPreflight.enabled=true` to force the cutoff logic. The logs will show `"quota_exhausted"` when a candidate is blocked due to insufficient quota.

## Diagnose Fallback and Retry Behavior

When a target returns a non-2xx status, the combo engine enters the retry loop managed by `dispatchWithCooldownRetry`. Look for the log entry *"All targets failed — retrying set"* around lines ~1240-1246 to identify when the engine cycles to the next retry set.

Each retry attempt logs the current count against `maxSetRetries` and the `retryDelayMs` value from the combo configuration. If all targets fail across all retry sets, the engine returns an `unavailableResponse` or `errorResponse` (lines ~620-640).

## Query Runtime Metrics and Health

### Access Combo Metrics Database

After execution, `recordComboRequest` in [`open-sse/services/comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/comboMetrics.ts) persists data to the `combo_metrics` SQLite table. You can query this data directly or use the CLI:

```bash
omniroute combo metrics get <combo-name>

```

The metrics include latency, token counts, success rates, and failure reasons.

### Run the Diagnostic CLI

Use the built-in health check to verify provider status and circuit breaker states:

```bash
omniroute doctor --output json

```

This command reports the status of rate limits, cooldown windows, and quota availability across all configured providers.

## Validate Request Entry Points

If a request never reaches the combo engine, verify it passed the **Zod validation** schemas in `src/app/api/v1/**/route.ts`. These files validate the request body and authentication before calling the service layer. Malformed requests are rejected with validation errors before [`combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/combo.ts) processes them.

## Code Examples

### Enabling Debug Logging and Reproducing Requests

```bash

# Start with maximum verbosity

LOG_LEVEL=debug omniroute start &

# Issue a failing request

curl -X POST http://localhost:20128/v1/chat/completions \
     -H "Authorization: Bearer <TOKEN>" \
     -H "Content-Type: application/json" \
     -d '{"model":"auto","messages":[{"role":"user","content":"Hello"}]}'

```

Expected log output includes:

```

COMBO  ▶️  routing directly to pinned context model: auto
COMBO  ⚙️  Weighted selection (sticky): 12 total targets
COMBO  ⏳  All targets failed — retrying set (1/3)
COMBO  ✅  Successful response from provider: openai/gpt-4o

```

### Inspecting Resolved Targets Programmatically

```typescript
import { resolveComboTargets, getComboFromData } from "./open-sse/services/combo.ts";

const combo = await getComboFromData("my-auto-combo");
const allCombos = []; // Fetch all combos if nested resolution is needed
const targets = await resolveComboTargets(combo, allCombos, 10);

console.log("Resolved targets:", targets.map(t => t.modelStr));

```

This expands wildcards via `expandProviderWildcardsInCombo` and returns the final routing list.

### Triggering Quota Preflight for Testing

```typescript
import { buildAutoCandidates } from "./open-sse/services/combo.ts";

const candidates = await buildAutoCandidates(
  resolvedTargets,
  "my-auto-combo",
  "session-123",
  undefined, // default reset-window config
  null       // default resilience settings
);

candidates.forEach(c => {
  console.log(`${c.provider}/${c.model} – quota ${c.quotaRemaining}%`);
});

```

## Key Files for Debugging

| File | Purpose |
|------|---------|
| [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) | Core combo handling, routing strategies, and fallback logic |
| [`open-sse/services/comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/comboMetrics.ts) | Persists latency, token usage, and success/failure counts |
| [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts) | Circuit breaker and provider cooldown helpers |
| [`open-sse/services/rateLimitSemaphore.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/rateLimitSemaphore.ts) | Global rate limiting state |
| [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts) | Per-provider cooldown tracking |
| [`open-sse/services/quotaPreflight.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/quotaPreflight.ts) | Hard quota cutoff implementation |
| [`src/lib/db/comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/db/comboMetrics.ts) | SQLite schema for metrics storage |
| [`src/lib/events/eventBus.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/src/lib/events/eventBus.ts) | Central event emitter for debug events |

## Summary

- **Enable `LOG_LEVEL=debug`** to see the combo engine's routing decisions and skip reasons.
- **Trace execution** through [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts), starting with `resolveComboTargets` (lines ~331-350).
- **Check `isTargetSelectableForWeighted`** (lines ~445-560) to identify why providers are skipped due to cooldowns, circuit breakers, or locks.
- **Inspect quota logic** in `buildAutoCandidates` (lines ~2910-3000) when candidates are filtered due to quota limits.
- **Use `omniroute doctor`** for a snapshot of provider health and circuit breaker states.
- **Validate entry points** in `src/app/api/v1/**/route.ts` if requests fail before reaching the service layer.

## Frequently Asked Questions

### Why is my request returning a 503 Service Unavailable error?

A 503 typically indicates that the **combo engine** exhausted all retry sets without finding a healthy provider. Check [`open-sse/services/combo.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/combo.ts) around lines ~1240-1260 for the *"All targets failed"* log entry. This occurs when all candidates are in cooldown, circuit breakers are open, or quotas are exhausted. Run `omniroute doctor` to verify provider health and circuit breaker states.

### How do I see which provider was selected for my request?

Look for the `COMBO` log entries that show the resolved target list after `resolveComboTargets` executes (lines ~331-350). If using weighted selection, the logs will indicate the chosen provider after `isTargetSelectableForWeighted` filters the list. For sticky routing, the logs will show the pinned context model being reused.

### Where are combo execution metrics stored?

Metrics are stored in a SQLite database via `recordComboRequest` in [`open-sse/services/comboMetrics.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/comboMetrics.ts). You can query the `combo_metrics` table directly or use the CLI command `omniroute combo metrics get <combo-name>`. The data includes request latency, token usage, and success rates for each combo execution.

### How can I check if a provider is in cooldown?

Provider cooldown state is tracked in [`open-sse/services/providerCooldownTracker.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/providerCooldownTracker.ts). When debugging, look for log entries containing `"provider in cooldown"` in the output from `isTargetSelectableForWeighted` (lines ~445-560). You can also check the global cooldown state via `omniroute doctor`, which reports cooldown windows and remaining seconds for each provider.