How OmniRoute Handles Retries with Backoff for Upstream Requests

OmniRoute implements a centralized retry mechanism using exponential backoff through the withRetry utility in src/lib/proxySubscription/fetchRetry.ts, automatically spacing retry attempts by doubling the delay after each transient failure while respecting provider Retry-After headers and configurable maximum ceilings.

OmniRoute treats every call to upstream LLM providers as a potentially transient operation requiring resilient fault tolerance. The framework automatically retries failed requests using an exponential backoff strategy that is centralized, configurable, and applied consistently across proxy subscriptions, token refreshes, and circuit breakers according to the diegosouzapw/OmniRoute source code.

Core Retry Architecture with withRetry

The foundation of OmniRoute's resilience strategy lives in src/lib/proxySubscription/fetchRetry.ts, which exports the generic withRetry<T>() function. This utility wraps any async operation, detects transient failures, and schedules subsequent attempts using an exponential delay calculation.

Rather than scattering retry logic throughout the codebase, OmniRoute centralizes the implementation so that proxy subscriptions, token health checks, and circuit breakers all invoke the same backoff algorithm. The function accepts a callback returning a Promise and automatically determines whether an error warrants a retry based on status codes and network conditions.

Detecting Transient Failures

The retry logic classifies specific error conditions as transient and therefore eligible for backoff. According to the implementation in src/lib/proxySubscription/fetchRetry.ts, the system retries on HTTP status codes 502, 503, 504, and 429, plus network-level errors that indicate temporary unavailability. Non-transient errors fail immediately without consuming retry attempts.

Exponential Backoff Algorithm

OmniRoute calculates retry delays using a standard exponential backoff formula with configurable guardrails. The algorithm executes these steps for every retry attempt:

1. Classify the error – Verify the failure is transient (5xx, 429, or network timeout).
2. Determine base delay – Use the provider's minRetryCooldownMs (default 500 ms) or extract a Retry-After header value when present.
3. Apply exponential multiplier – Calculate delay = baseDelay * Math.pow(2, attempt - 1), doubling the wait time with each attempt.
4. Enforce ceiling – Clamp the result against maxBackoffMs to prevent runaway wait times.
5. Schedule execution – Pause for the computed duration before invoking the next attempt, up to maxRetries (typically 3–5).

If the upstream response includes a Retry-After header, OmniRoute honors this value over the calculated exponential delay. As implemented in open-sse/services/accountFallback.ts (lines 728–755), the header value wins unconditionally, allowing the system to respect the provider's explicit throttling window.

Connection-Level Cooldown Calculations

For per-connection failure tracking, open-sse/services/accountFallback.ts provides calculateExponentialBackoffCooldown(). This helper computes delays using minRetryCooldownMs * 2^(failures - 1) and respects the maxCooldownMs threshold defined in src/lib/resilience/settings/types.ts. This ensures individual connection failures do not impact the global retry policy while still preventing thundering herds.

Implementation Across the Request Lifecycle

OmniRoute wires the retry utility into several critical paths to ensure consistent behavior regardless of which upstream provider is being contacted.

Proxy Subscription Handling

The subscription service in src/lib/proxySubscription/subscriptionService.ts wraps all upstream fetches using withRetry(). At line 379, the service passes its doSafeFetch function into the retry wrapper, ensuring that streaming subscriptions benefit from automatic recovery without dropping client connections during transient provider outages.

Token Refresh Operations

Credential refreshes use the same exponential backoff strategy to avoid hammering authentication endpoints. The open-sse/services/tokenRefresh/circuitBreaker.ts module exports refreshWithRetry(), which applies the backoff logic when fetching new tokens. Additionally, src/lib/tokenHealthCheck.ts implements retry policies specifically for background token validation tasks.

Provider Circuit Breakers

At the provider level, src/shared/utils/circuitBreaker.ts tracks consecutive failures and transitions between CLOSED, OPEN, and HALF-OPEN states. When the breaker enters the OPEN state, it applies an exponential multiplier (default ×16) to the reset timeout, effectively backing off the entire provider rather than individual requests. This prevents OmniRoute from repeatedly attempting requests to a completely unavailable upstream service.

Configuration and Customization

All backoff parameters are centralized in src/lib/resilience/settings/types.ts, allowing operators to adjust defaults globally or override values per provider via src/shared/constants/providers.ts. Key configuration options include:

• minRetryCooldownMs – Base delay for the first retry (default 500 ms)
• maxBackoffMs – Hard ceiling for any delay calculation
• maxRetries – Maximum number of attempts before hard failure

Code Examples

Wrapping a Subscription Fetch

import { withRetry } from "@/lib/proxySubscription/fetchRetry";

async function fetchData(url: string, headers: HeadersInit) {
  return withRetry(() => doSafeFetch(url, headers));
}

Source: src/lib/proxySubscription/subscriptionService.ts (line 379)

Token Refresh with Backoff

import { refreshWithRetry } from "open-sse/services/tokenRefresh/circuitBreaker";

const refreshed = await refreshWithRetry(providerId, async () => {
  return await fetchTokenEndpoint(...);
});

Source: open-sse/services/tokenRefresh/circuitBreaker.ts (line 114)

Manual Backoff Calculation

import { calculateExponentialBackoffCooldown } from "open-sse/services/accountFallback";

const cooldown = calculateExponentialBackoffCooldown({
  failures: 3,
  minRetryCooldownMs: 500,
  maxCooldownMs: 30_000,
});

Source: open-sse/services/accountFallback.ts (line 1546)

Key Implementation Files

• src/lib/proxySubscription/fetchRetry.ts – Core withRetry implementation and exponential backoff logic
• src/lib/proxySubscription/subscriptionService.ts – Integration point for streaming subscriptions
• open-sse/services/accountFallback.ts – Per-connection cooldown calculations and Retry-After header handling
• src/shared/utils/circuitBreaker.ts – Provider-level circuit breaker with exponential multipliers
• src/lib/resilience/settings/types.ts – Configuration types for backoff min/max values
• open-sse/services/tokenRefresh/circuitBreaker.ts – Token refresh retry wrapper
• src/lib/tokenHealthCheck.ts – Background token validation retry strategy

Summary

• OmniRoute centralizes retry logic in the withRetry utility within src/lib/proxySubscription/fetchRetry.ts, ensuring consistent behavior across proxy subscriptions and token operations.
• The exponential backoff algorithm doubles the delay after each failure using baseDelay * 2^(attempt-1) while respecting configurable minimums and maximums defined in src/lib/resilience/settings/types.ts.
• Transient error detection covers HTTP 5xx status codes, 429 rate limits, and network failures, immediately failing on permanent errors to conserve resources.
• The system respects upstream Retry-After headers when present, overriding the exponential calculation to comply with provider-specific throttling windows.
• Circuit breakers in src/shared/utils/circuitBreaker.ts provide provider-level protection using exponential multipliers, while connection-level backoff in open-sse/services/accountFallback.ts isolates failures to specific upstream links.

Frequently Asked Questions

What HTTP status codes trigger a retry in OmniRoute?

OmniRoute retries requests that return 502 Bad Gateway, 503 Service Unavailable, 504 Gateway Timeout, or 429 Too Many Requests, along with network-level errors indicating temporary unavailability. Permanent errors such as 400 Bad Request or 401 Unauthorized fail immediately without retry attempts, as these indicate client-side issues that will not resolve with time.

How does OmniRoute handle rate-limiting headers from upstream providers?

When an upstream response includes a Retry-After header, OmniRoute extracts this value and uses it as the delay duration for the next retry attempt, bypassing the exponential calculation. This logic is implemented in open-sse/services/accountFallback.ts (lines 728–755), ensuring the framework respects the provider's explicit throttling instructions rather than guessing the appropriate backoff timing.

Can I customize the retry and backoff behavior for specific providers?

Yes. Omniroute exposes retry configuration through src/lib/resilience/settings/types.ts, where you can define minRetryCooldownMs, maxBackoffMs, and maxRetries values. Per-provider overrides can be specified in src/shared/constants/providers.ts, allowing different upstream services to have unique retry policies based on their reliability characteristics or contractual rate limits.

What is the difference between connection-level and provider-level backoff?

Connection-level backoff, calculated by calculateExponentialBackoffCooldown() in open-sse/services/accountFallback.ts, tracks failures for individual upstream connections and applies exponential delays using minRetryCooldownMs * 2^(failures-1). Provider-level backoff, managed by src/shared/utils/circuitBreaker.ts, tracks failures across all connections to a specific provider and applies an exponential multiplier (default ×16) to the circuit reset timeout when the breaker opens, protecting the entire service from repeatedly attempting requests to a completely failed provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}