# How FreeLLMAPI Implements Key Rotation for Load Distribution in Fusion Panels

> Discover how FreeLLMAPI uses key rotation with skip-sets and cooldowns for efficient load distribution and fallback strategies, ensuring optimal API performance.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-07-01

---

**FreeLLMAPI distributes API traffic by rotating through up to four keys per model slot, using skip-sets to blacklist throttled credentials and applying cooldowns before falling back to overflow models.**

The Fusion feature in tashfeenahmed/freellmapi orchestrates multiple LLM models simultaneously to generate synthesized or judged responses. To prevent a single throttled API key from collapsing the entire panel, the system implements **key rotation** that cycles through available credentials within each pinned slot. This mechanism balances load across a model's entire key pool while maintaining strict attempt limits to ensure fast failures.

## Per-Slot Key Rotation Budgets

FreeLLMAPI caps the number of key attempts per panel slot to prevent stalled requests. This budget is enforced by a constant in the Fusion service.

### MAX_SLOT_ATTEMPTS Configuration

In [`server/src/services/fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/fusion.ts), the system defines a hard limit for key rotation:

```typescript
// server/src/services/fusion.ts
// Per-slot key-rotation budget: a slot tries at most this many keys of its
// pinned model before it's dropped. Small — a model with every key cooled down
// should fail fast, not stall the whole panel.
const MAX_SLOT_ATTEMPTS = 4;         // ← key-rotation limit
const MAX_JUDGE_ATTEMPTS = 6;        // (judge side)

```

This constant ensures that if every key for a specific model is rate-limited or invalid, the slot fails within four attempts rather than blocking the panel indefinitely.

## Iterative Call Logic with Skip-Sets

The `runModelCall` function drives the rotation mechanism by maintaining exclusion sets that track unusable resources across retries.

### Skip-Set Management

The function initializes two `Set` objects to filter routing decisions:

- **`skipKeys`** – Stores individual API keys that have hit rate limits, payment errors, or authentication failures.
- **`skipModels`** – Stores entire model IDs that return permanent errors (e.g., HTTP 404).

```typescript
async function runModelCall(..., maxAttempts) {
  const skipKeys = new Set<string>();
  const skipModels = new Set<number>();
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const route = getRoute(skipKeys, skipModels);
    // ... execute call ...
    if (empty || error) {
      // record failure, add key to skip list, apply cooldown
      skipKeys.add(`${route.platform}:${route.modelId}:${route.keyId}`);
      setCooldown(...);
      continue;                 // retry with a different key
    }
    // ...
  }
}

```

As implemented in the source code at lines 10–27 and 70–73 of [`server/src/services/fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/fusion.ts), the router's `getRoute` function honors these skip-sets by excluding blacklisted keys from selection. When a call fails, the offending key enters a cooldown period, and the loop immediately retries with the next available credential.

## Panel-Wide Failover Behavior

When a slot exhausts its `MAX_SLOT_ATTEMPTS` budget without success, FreeLLMAPI treats that slot as failed. The `selectPanel` logic then pulls a replacement model from an **overflow** queue to maintain the configured panel size. This design isolates key-rotation failures to individual slots while preserving the overall parallelism of the Fusion request.

## Judge Model Rotation

The same rotation pattern applies to the optional judge model used for ranking or synthesizing responses, but with a higher tolerance for transient failures. The system uses `MAX_JUDGE_ATTEMPTS = 6` to give the judge slot additional retry capacity before falling back to alternative evaluation strategies.

## Practical Implementation Example

Key rotation occurs automatically within the Fusion execution flow. Callers do not manually manage credentials; the system handles rotation transparently:

```typescript
import { runFusion } from './server/src/services/fusion.js';
import { CompletionOptions } from './server/src/providers/base.js';

const result = await runFusion({
  messages: [{ role: 'user', content: 'Explain quantum tunneling.' }],
  config: { k: 4, strategy: 'synthesize' },   // Fusion config
  options: { temperature: 0.7 } as CompletionOptions,
  estimatedTokens: 100,
  hooks: {
    onPanel: info => console.log('Panel slot:', info),
    onJudge: info => console.log('Judge:', info),
  },
});

console.log(result.response.choices[0].message.content);

```

Behind this interface, `runModelCall` automatically attempts up to `MAX_SLOT_ATTEMPTS` keys for each panel member, cycling through the model's available credentials until success or exhaustion.

## Summary

- **FreeLLMAPI** pins each Fusion panel slot to a specific model but rotates through multiple API keys belonging to that model.
- **`MAX_SLOT_ATTEMPTS = 4`** limits retries per slot to prevent throttled keys from stalling the entire panel.
- **`skipKeys` and `skipModels`** sets permanently exclude failed credentials from the routing pool on subsequent attempts.
- **Cooldown periods** are applied to failed keys before they become eligible for future requests.
- **Overflow logic** replaces exhausted slots with alternative models to maintain panel stability.
- **Judge models** use a higher budget (`MAX_JUDGE_ATTEMPTS = 6`) to tolerate transient failures during response evaluation.

## Frequently Asked Questions

### How many API keys will FreeLLMAPI try for a single model request?

FreeLLMAPI attempts up to **four keys** per model slot (`MAX_SLOT_ATTEMPTS = 4`) before marking that slot as failed and potentially substituting it with an overflow model. The judge model uses a separate budget of six attempts.

### What happens when an API key hits a rate limit during Fusion?

The system adds the key to the `skipKeys` set, applies a cooldown period via `setCooldown()`, and immediately retries the request with the next available key from the same model's pool. The blacklisted key is excluded from all subsequent routing decisions within that Fusion call.

### Where is the key rotation logic implemented in the codebase?

The core rotation mechanism resides in [`server/src/services/fusion.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/fusion.ts), specifically within the `runModelCall` function (lines 10–73). The routing logic that respects skip-sets is implemented in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), which provides `getRoute` and related methods that filter out excluded keys and models.

### Does the caller need to handle key rotation manually?

No. Key rotation is fully automatic when using `runFusion`. The caller provides the messages and configuration, while the Fusion service internally manages the `skipKeys` tracking, cooldown application, and fallback to overflow models.