# How OmniRoute's Model Lockout Feature Prevents Repeated Failures

> OmniRoute's model lockout stops repeated failures by temporarily excluding failing provider/model pairs from the candidate pool. Learn how it works.

- Repository: [Diego Rodrigues de Sa e Souza/OmniRoute](https://github.com/diegosouzapw/OmniRoute)
- Tags: feature-explanation
- Published: 2026-08-06

---

**OmniRoute prevents repeated failures by recording a temporary model lockout that excludes failing provider/model pairs from the candidate pool until their cooldown expires.**

OmniRoute's **model lockout** system is a granular resilience mechanism that isolates problematic AI models without penalizing entire provider connections. Implemented in the [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) service, this feature tracks failure patterns, applies intelligent cooldowns, and automatically decays lockouts to restore healthy traffic flow.

---

## Detecting Failures That Trigger Lockouts

The lockout process begins when an upstream request returns a status code ≥ 400 with a classified error. In [`open-sse/services/accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/open-sse/services/accountFallback.ts), the `recordModelLockoutFailure` function handles this detection:

```typescript
// Lines 604-610 in accountFallback.ts
// Called when errors like 'rate_limit' or 'quota_exhausted' occur
recordModelLockoutFailure(
  provider: string,
  connectionId: string,
  model: string,
  reason: string,
  status: number,
  fallbackCooldownMs: number,
  profile?: ProviderProfile,
  options?: { exactCooldownIsUpstreamReset?: boolean }
)

```

The function captures critical context: which **provider**, **connection**, and **model** failed, plus the specific **failure reason** that determines how the cooldown is calculated.

---

## Building Unique Lock Keys

Each lockout receives a deterministic identifier to prevent collisions and enable precise targeting. The lock key construction appears at lines 632-635:

```typescript
// Standard key format: provider:connectionId:model
const lockKey = `${provider}:${connectionId}:${model}`;

// For quota-family providers, an "exact" key may be used
const exactKey = `${provider}:${connectionId}:${model}:exact`;

```

This string-based keying ensures that lockouts are **scoped to specific model instances** rather than applied broadly across unrelated workloads.

---

## Computing Intelligent Cooldown Periods

OmniRoute calculates cooldowns through two complementary strategies in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts):

### Explicit Upstream Resets

When providers supply a `Retry-After` header or similar signal, `selectLockoutCooldownMs` (lines 94-102) respects that exact duration:

```typescript
// Use the upstream's requested reset time
const cooldownMs = selectLockoutCooldownMs(
  reason,
  upstreamResetTimestamp,  // From Retry-After or API response
  profile
);

```

### Exponential Back-Off

Without explicit guidance, `getScaledCooldown` (lines 68-74) applies failure-count-based scaling:

```typescript
// Back-off increases with repeated failures
const cooldownMs = getScaledCooldown(
  baseCooldown,
  failureCount,
  profile?.resilience?.backoffMultiplier
);

```

The combination ensures **fast recovery for transient errors** and **aggressive throttling for persistent problems**.

---

## Persisting Lockout State

Lockout entries are stored in an in-memory `modelLockouts` map with full audit context. The `ModelLockoutEntry` type (lines 86-93) captures:

```typescript
interface ModelLockoutEntry {
  reason: string;        // e.g., 'quota_exhausted', 'rate_limit'
  until: number;         // Unix timestamp when lock expires
  lockedAt: number;      // When lock was created
  failureCount: number;  // Cumulative failures for this key
  cooldownMs: number;    // Applied cooldown duration
  isExact: boolean;      // Whether this is a quota-family exact lock
}

```

This structured storage enables both **runtime decisions** and **operational visibility** without external database dependencies.

---

## Applying Granular Locks

OmniRoute distinguishes between **per-model quotas** and **connection-wide limits** through conditional logic at lines 223-238:

### Per-Model Quota Providers

For providers flagging `hasPerModelQuota`, `lockModelIfPerModelQuota` isolates only the offending model:

```typescript
// Lock applies to this specific model only
lockModelIfPerModelQuota(provider, connectionId, model, entry);
// Other models on same connection remain available

```

### Standard Providers

Without per-model quota support, the lock functions as a **connection-level cooldown**, protecting upstream resources while minimizing blast radius.

---

## Enforcing Locks During Request Routing

The critical enforcement point occurs during credential selection. The `isModelLocked` function (lines 220-227) filters the candidate pool:

```typescript
// Before selecting credentials for a request
if (isModelLocked(provider, connectionId, model)) {
  // Skip this model—it's under active lockout
  continue; // Move to next candidate
}

```

This **pre-selection filtering** guarantees that locked models never receive traffic, eliminating wasted requests and repeated failures.

---

## Automatic Decay and Recovery

Lockouts self-heal through two mechanisms (lines 86-96):

1. **Explicit expiration** — locks are automatically removed when `until` timestamp passes
2. **Failure count decay** — `decayModelFailureCount` reduces historical failure weight over time, preventing permanent blacklisting

```typescript
// Gradual forgiveness for stabilized models
decayModelFailureCount(lockKey, decayFactor);

```

This **adaptive recovery** ensures models can re-enter rotation once conditions improve.

---

## Operational Visibility

Administrators monitor lockout state through dedicated query functions (lines 336-354):

| Function | Purpose |
|----------|---------|
| `getModelLockoutInfo(provider, connectionId, model)` | Check single model status |
| `getAllModelLockouts()` | List all active lockouts for dashboard display |

The [`providerHealthAutopilot.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/providerHealthAutopilot.ts) module surfaces this data for real-time monitoring and manual lockout clearance when needed.

---

## Summary

- **Detection**: `recordModelLockoutFailure` captures classified errors (≥400 status) with full context
- **Identification**: Unique `provider:connectionId:model` keys enable precise targeting
- **Cooldown**: Respects upstream `Retry-After` or applies exponential back-off via `getScaledCooldown`
- **Storage**: `ModelLockoutEntry` structs in `modelLockouts` map capture reason, expiration, and failure history
- **Enforcement**: `isModelLocked` filters failing models before request assignment
- **Recovery**: Automatic expiration and failure count decay restore healthy models
- **Visibility**: `getAllModelLockouts` exposes state for operational dashboards

---

## Frequently Asked Questions

### What types of errors trigger a model lockout?

Lockouts activate on HTTP status ≥400 when accompanied by classified reasons like `rate_limit`, `quota_exhausted`, or `invalid_auth`. The `recordModelLockoutFailure` function in [`accountFallback.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/accountFallback.ts) only creates entries for explicitly categorized failures, ensuring transient network errors don't unnecessarily disable models.

### How does OmniRoute handle providers with per-model quotas?

For providers advertising `hasPerModelQuota`, `lockModelIfPerModelQuota` applies the lock exclusively to the failing model. Other models on the same connection continue serving traffic. Without this flag, the cooldown applies connection-wide as a protective measure.

### Can operators manually clear a model lockout?

Yes. The [`providerHealthAutopilot.ts`](https://github.com/diegosouzapw/OmniRoute/blob/main/providerHealthAutopilot.ts) monitoring module exposes administrative endpoints for lockout inspection and manual clearance. Operators can view active lockouts via `getAllModelLockouts` and remove entries when confident the underlying issue is resolved.

### What prevents permanent blacklisting of recovered models?

The `decayModelFailureCount` mechanism gradually reduces historical failure counts, and all locks carry explicit `until` timestamps. Once expired, models automatically return to the candidate pool with refreshed back-off calculations, ensuring no model remains locked indefinitely.