How OmniRoute's Rate Limit Manager Handles Headers from Different Providers

OmniRoute's rate limit manager inspects HTTP response headers from each provider to dynamically adjust local Bottleneck queue settings, using provider-specific header maps for Anthropic and standard providers while parsing reset times and handling 429 errors through limiter eviction rather than shutdown.

OmniRoute (diegosouzapw/OmniRoute) protects API integrations with an adaptive rate-limiting layer built on the Bottleneck queue library. The rate limit manager automatically learns each provider's rate-limiting contract by inspecting response headers after every request, dynamically adjusting throttle settings to prevent 429 errors while maximizing throughput.

Provider-Specific Header Mapping

The manager selects between two distinct header schemas based on the provider identifier. In open-sse/services/rateLimitManager.ts, the updateFromHeaders function chooses between STANDARD_HEADERS (used by most providers) and ANTHROPIC_HEADERS (specific to Claude/Anthropic) using a simple conditional check:

const headerMap =
  provider === "claude" || provider === "anthropic"
    ? ANTHROPIC_HEADERS
    : STANDARD_HEADERS;

This selection occurs at lines 122-125 of rateLimitManager.ts, ensuring that provider-specific field names (such as Anthropic's anthropic-ratelimit-requests-remaining versus standard x-ratelimit-remaining) are mapped correctly before parsing.

Normalizing and Parsing Header Values

Once the appropriate header map is selected, the manager normalizes the HTTP Headers object using toPlainHeaders to create a lower-cased key map. It then extracts the critical rate-limiting fields:

const limit       = parseInt(getHeader(headerMap.limit));
const remaining   = parseInt(getHeader(headerMap.remaining));
const resetStr    = getHeader(headerMap.reset);
const retryAfter  = getHeader(headerMap.retryAfter);

This extraction happens at lines 126-134 of rateLimitManager.ts. The normalization ensures case-insensitive header matching, accommodating providers that may send X-RateLimit-Remaining or x-ratelimit-remaining.

Flexible Reset Time Parsing

Provider responses vary wildly in how they express reset intervals—some send Unix timestamps, others send ISO dates, and some use human-readable strings like 1m30s. The utility function parseResetTime in open-sse/services/rateLimitManager/headers.ts (lines 30-56) handles this heterogeneity:

export function parseResetTime(value) { … }

The function always converts the input to a millisecond interval, allowing the manager to schedule accurate retry delays regardless of the provider's chosen format.

Handling Rate Limit Violations (429 Responses)

When a provider returns HTTP 429 (Too Many Requests), the manager implements an eviction strategy rather than permanently halting the limiter. At lines 136-152 of rateLimitManager.ts, the code extracts a retry interval (defaulting to 60 seconds), deletes the existing limiter instance, and calls limiter.disconnect():

if (status === 429) {
  const retryAfterMs = parseResetTime(retryAfterStr) || 60000;
  limiters.delete(limiterKey);
  …
  trackAsyncOperation(limiter.disconnect());
}

Critical implementation detail: The manager never calls limiter.stop(), which would permanently reject future jobs. Instead, disconnect() frees the internal heartbeat timer while allowing the next request to spawn a fresh limiter instance.

Soft Limits and Over-Limit Signals

Some providers (such as Fireworks) send proactive warnings via headers like x-ratelimit-over-limit: yes before enforcing hard limits. The manager treats these as soft-limit signals, injecting a modest 200ms delay to slow traffic without stopping it entirely. This adjustment occurs at lines 162-170 of rateLimitManager.ts:

if (overLimit === "yes") {
  limiter.updateSettings({ minTime: 200 });
}

Adaptive Throttling on Successful Requests

For normal successful responses containing rate-limit metadata, the manager calculates an optimal minTime (minimum time between requests) using the formula:

const minTime = Math.max(0, Math.floor(60000 / limit) - 10);

This yields approximately one request per minute divided by the provider's stated limit, minus a small buffer. Additionally, the manager adjusts the Bottleneck reservoir based on remaining quota:

• Low quota (<10% of limit): Throttle aggressively using reservoir constraints
• High quota (>50% of limit): Relax constraints to maximize throughput

These dynamic updates occur at lines 174-197 of rateLimitManager.ts.

Persisting Learned Limits

After adjusting the limiter, the manager records the discovered values in learnedLimits and persists them to the settings table with debouncing. This ensures that rate-limit knowledge survives application restarts. The persistence call at lines 201-209 of rateLimitManager.ts looks like:

recordLearnedLimit(provider, connectionId, { limit, remaining, minTime }, model);

Implementation Example

To leverage this system, wrap provider-specific executor calls with withRateLimit, then feed the response headers to updateFromHeaders:

import { withRateLimit, updateFromHeaders } from "@/open-sse/services/rateLimitManager";
import { execute } from "@/open-sse/executors/default";

// Queue the request until the local limiter grants a slot
const response = await withRateLimit(
  "openai",
  conn.id,
  model,
  () => execute(request)
);

// Learn from the provider's response headers
updateFromHeaders(
  "openai",
  conn.id,
  response.headers,
  response.status,
  model
);

The withRateLimit call blocks until the local Bottleneck queue grants a slot. Once the upstream request completes, updateFromHeaders ingests the provider's rate-limit headers and instantly re-configures the limiter for subsequent requests.

Summary

• Provider-specific schemas: OmniRoute switches between STANDARD_HEADERS and ANTHROPIC_HEADERS based on the provider ID to correctly map header fields.
• Flexible parsing: The parseResetTime utility handles multiple timestamp formats, converting them to millisecond intervals for consistent scheduling.
• 429 recovery: The manager evicts and disconnect()s limiters on 429 errors rather than calling stop(), allowing fresh limiters to replace them on the next request.
• Soft-limit handling: Warning headers like x-ratelimit-over-limit trigger modest 200ms throttling to prevent hitting hard limits.
• Adaptive tuning: Normal responses update minTime calculations (≈60,000÷limit) and reservoir settings based on remaining quota percentages.
• Persistence: Learned limits are debounced and saved to survive application restarts.

Frequently Asked Questions

How does OmniRoute distinguish between Anthropic and standard provider headers?

The manager checks the provider identifier at runtime. If the provider string equals "claude" or "anthropic", it selects the ANTHROPIC_HEADERS map; otherwise, it defaults to STANDARD_HEADERS. This selection happens in open-sse/services/rateLimitManager.ts at lines 122-125, ensuring correct field mapping for Anthropic's unique header naming conventions.

What happens when a provider returns a 429 status?

When encountering HTTP 429, the manager extracts a retry-after value (defaulting to 60 seconds), deletes the current limiter instance from the internal limiters Map, and calls limiter.disconnect() to free resources without permanently rejecting future jobs. This eviction strategy allows the next request to create a fresh limiter, effectively pausing the connection for the retry interval then resuming with clean state.

How does the manager handle ambiguous reset time formats?

The parseResetTime function in open-sse/services/rateLimitManager/headers.ts handles inputs ranging from simple integers (seconds), ISO 8601 dates, Unix timestamps, and human-readable strings (like 1m30s). It normalizes all formats to millisecond intervals, ensuring the scheduler can accurately calculate retry delays regardless of the provider's chosen representation.

Why does OmniRoute use disconnect() instead of stop() for rate limiters?

stop() permanently halts the Bottleneck instance and rejects all pending and future jobs, which would break the connection indefinitely. disconnect() only frees the internal heartbeat timer and stops the limiter from accepting new jobs while allowing the instance to be garbage collected. This permits the manager to evict the old limiter and instantiate a fresh one on the next request, achieving a temporary pause rather than a permanent shutdown.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}