How FreeLLMAPI Implements Key Rotation for Load Distribution in Fusion Panels

FreeLLMAPI distributes API traffic by rotating through up to four keys per model slot, using skip-sets to blacklist throttled credentials and applying cooldowns before falling back to overflow models.

The Fusion feature in tashfeenahmed/freellmapi orchestrates multiple LLM models simultaneously to generate synthesized or judged responses. To prevent a single throttled API key from collapsing the entire panel, the system implements key rotation that cycles through available credentials within each pinned slot. This mechanism balances load across a model's entire key pool while maintaining strict attempt limits to ensure fast failures.

Per-Slot Key Rotation Budgets

FreeLLMAPI caps the number of key attempts per panel slot to prevent stalled requests. This budget is enforced by a constant in the Fusion service.

MAX_SLOT_ATTEMPTS Configuration

In server/src/services/fusion.ts, the system defines a hard limit for key rotation:

// server/src/services/fusion.ts
// Per-slot key-rotation budget: a slot tries at most this many keys of its
// pinned model before it's dropped. Small — a model with every key cooled down
// should fail fast, not stall the whole panel.
const MAX_SLOT_ATTEMPTS = 4;         // ← key-rotation limit
const MAX_JUDGE_ATTEMPTS = 6;        // (judge side)

This constant ensures that if every key for a specific model is rate-limited or invalid, the slot fails within four attempts rather than blocking the panel indefinitely.

Iterative Call Logic with Skip-Sets

The runModelCall function drives the rotation mechanism by maintaining exclusion sets that track unusable resources across retries.

Skip-Set Management

The function initializes two Set objects to filter routing decisions:

• skipKeys – Stores individual API keys that have hit rate limits, payment errors, or authentication failures.
• skipModels – Stores entire model IDs that return permanent errors (e.g., HTTP 404).

async function runModelCall(..., maxAttempts) {
  const skipKeys = new Set<string>();
  const skipModels = new Set<number>();
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const route = getRoute(skipKeys, skipModels);
    // ... execute call ...
    if (empty || error) {
      // record failure, add key to skip list, apply cooldown
      skipKeys.add(`${route.platform}:${route.modelId}:${route.keyId}`);
      setCooldown(...);
      continue;                 // retry with a different key
    }
    // ...
  }
}

As implemented in the source code at lines 10–27 and 70–73 of server/src/services/fusion.ts, the router's getRoute function honors these skip-sets by excluding blacklisted keys from selection. When a call fails, the offending key enters a cooldown period, and the loop immediately retries with the next available credential.

Panel-Wide Failover Behavior

When a slot exhausts its MAX_SLOT_ATTEMPTS budget without success, FreeLLMAPI treats that slot as failed. The selectPanel logic then pulls a replacement model from an overflow queue to maintain the configured panel size. This design isolates key-rotation failures to individual slots while preserving the overall parallelism of the Fusion request.

Judge Model Rotation

The same rotation pattern applies to the optional judge model used for ranking or synthesizing responses, but with a higher tolerance for transient failures. The system uses MAX_JUDGE_ATTEMPTS = 6 to give the judge slot additional retry capacity before falling back to alternative evaluation strategies.

Practical Implementation Example

Key rotation occurs automatically within the Fusion execution flow. Callers do not manually manage credentials; the system handles rotation transparently:

import { runFusion } from './server/src/services/fusion.js';
import { CompletionOptions } from './server/src/providers/base.js';

const result = await runFusion({
  messages: [{ role: 'user', content: 'Explain quantum tunneling.' }],
  config: { k: 4, strategy: 'synthesize' },   // Fusion config
  options: { temperature: 0.7 } as CompletionOptions,
  estimatedTokens: 100,
  hooks: {
    onPanel: info => console.log('Panel slot:', info),
    onJudge: info => console.log('Judge:', info),
  },
});

console.log(result.response.choices[0].message.content);

Behind this interface, runModelCall automatically attempts up to MAX_SLOT_ATTEMPTS keys for each panel member, cycling through the model's available credentials until success or exhaustion.

Summary

• FreeLLMAPI pins each Fusion panel slot to a specific model but rotates through multiple API keys belonging to that model.
• MAX_SLOT_ATTEMPTS = 4 limits retries per slot to prevent throttled keys from stalling the entire panel.
• skipKeys and skipModels sets permanently exclude failed credentials from the routing pool on subsequent attempts.
• Cooldown periods are applied to failed keys before they become eligible for future requests.
• Overflow logic replaces exhausted slots with alternative models to maintain panel stability.
• Judge models use a higher budget (MAX_JUDGE_ATTEMPTS = 6) to tolerate transient failures during response evaluation.

Frequently Asked Questions

How many API keys will FreeLLMAPI try for a single model request?

FreeLLMAPI attempts up to four keys per model slot (MAX_SLOT_ATTEMPTS = 4) before marking that slot as failed and potentially substituting it with an overflow model. The judge model uses a separate budget of six attempts.

What happens when an API key hits a rate limit during Fusion?

The system adds the key to the skipKeys set, applies a cooldown period via setCooldown(), and immediately retries the request with the next available key from the same model's pool. The blacklisted key is excluded from all subsequent routing decisions within that Fusion call.

Where is the key rotation logic implemented in the codebase?

The core rotation mechanism resides in server/src/services/fusion.ts, specifically within the runModelCall function (lines 10–73). The routing logic that respects skip-sets is implemented in server/src/services/router.ts, which provides getRoute and related methods that filter out excluded keys and models.

Does the caller need to handle key rotation manually?

No. Key rotation is fully automatic when using runFusion. The caller provides the messages and configuration, while the Fusion service internally manages the skipKeys tracking, cooldown application, and fallback to overflow models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}