How the FreeLLMAPI Router Selects Providers for LLM Requests

The FreeLLMAPI router implements a deterministic, analytics-driven pipeline that filters providers by quota and rate limits, scores remaining candidates using multi-axis metrics including latency and success rates, and automatically selects the highest-ranked provider while maintaining a fallback chain for resilience.

FreeLLMAPI is an open-source API aggregation layer that intelligently routes requests to multiple large language model (LLM) providers. Understanding exactly how the FreeLLMAPI router selects providers for requests enables developers to optimize their integration and predict routing behavior. The system implements a sophisticated scoring engine that weighs real-time performance data against static configuration rules to minimize costs and maximize reliability.

The Seven-Stage Provider Selection Pipeline

The routing decision follows a strict pipeline defined in server/src/services/router.ts. Each stage eliminates unsuitable candidates or adjusts their priority scores before the final selection.

Model-Group Resolution and Fallback Chains

When a request arrives with model: "auto" or a specific group identifier, the router first expands the requested model identifier into candidate providers. According to comments around line 724 in router.ts, the system walks a fallback chain defined in the UI configuration, creating an ordered list of potential handlers. If the caller specifies an explicit provider such as openrouter/gpt-4o-mini, the router treats that provider as the sole primary candidate but still maintains the fallback chain for error recovery.

Provider-Quota Filtering

For each candidate model, the router checks the provider-quota table to verify whether the current API key possesses the necessary permissions. The mapping of platforms to quota keys resides in server/src/services/provider-quota.ts (lines 122-165). This step enforces free-only or paid-only restrictions, immediately removing providers that the key cannot access.

Rate-Limit and Penalty Inspection

The router consults the ratelimit and penalty-inspector services to identify blocked or throttled providers. If a key has exceeded a provider's Requests Per Day (RPD) limit, the system benchmarks that provider for 24 hours and skips it during selection. The rate-limit logic in server/src/services/ratelimit.ts (line 185) tracks these limits, while server/src/services/penalty-inspector.ts retrieves active penalties via the getAllPenalties function to downgrade providers with recent failures.

Multi-Axis Scoring System

Each remaining candidate receives a score vector calculated in server/src/services/scoring.ts. The system evaluates multiple axes:

• Latency: Historical response times
• Success-rate: Percentage of successful requests
• Token-budget: Cost efficiency metrics
• Custom weights: Runtime overrides set via administrative endpoints

The scoring engine supports multiple routing strategies, including tier-first (prioritizing provider tiers) and rank-as-tiebreaker (using rank to resolve score ties), as documented in the top comments of scoring.ts.

Custom Weights and Runtime Strategy

Administrators can modify router behavior at runtime through the declarative configuration API. The setCustomWeights and setRoutingStrategy functions in server/src/services/declarative-config.ts mutate in-memory weighting tables that the scorer reads immediately. This allows dynamic adjustment of provider priorities without restarting the service.

Final Selection and Error Handling

After scoring, the router selects the highest-ranked candidate. If the chosen provider returns a 429 error or other hard failure, the router transparently retries the request with the next-best candidate. The fallback walk logic appears around line 735 in router.ts, ensuring that requests eventually succeed even if primary providers fail.

Response Metric Recording

The HTTP entry point in server/src/routes/proxy.ts wraps the selected provider's response. It imports routeRequest to execute the routing decision and recordSuccess to feed performance data back into the scoring system, creating a self-healing feedback loop.

Practical Implementation Examples

Automatic Provider Selection

Trigger the auto-router to automatically select the best provider based on your API key's quotas and current performance metrics:

import axios from 'axios';

const response = await axios.post(
  'https://api.freellmapi.com/v1/chat/completions',
  {
    model: 'auto',
    messages: [{ role: 'user', content: 'Explain quantum tunneling.' }]
  },
  { headers: { Authorization: `Bearer ${YOUR_API_KEY}` } }
);

Explicit Provider Selection with Fallback

Force a specific provider while retaining the router's fallback capabilities:

const response = await axios.post(
  'https://api.freellmapi.com/v1/chat/completions',
  {
    model: 'openrouter/gpt-4o-mini',
    messages: [{ role: 'user', content: 'Generate a poem.' }]
  },
  { headers: { Authorization: `Bearer ${YOUR_API_KEY}` } }
);

Administrative Routing Configuration

Adjust the routing strategy at runtime using administrative privileges:

await axios.post(
  'https://api.freellmapi.com/v1/admin/routing/strategy',
  { strategy: 'rank' },
  { headers: { Authorization: `Bearer ${ADMIN_KEY}` } }
);

Summary

• The FreeLLMAPI router uses a deterministic seven-stage pipeline defined in server/src/services/router.ts to select providers.
• Quota filtering in provider-quota.ts and rate-limit checks in ratelimit.ts eliminate ineligible candidates before scoring.
• The scoring engine in scoring.ts ranks providers using latency, success rates, and custom weights according to configurable strategies.
• Automatic fallback occurs when the primary provider returns errors, with the retry logic implemented around line 735 of router.ts.
• Runtime configuration via declarative-config.ts allows dynamic adjustment of weights and strategies without service interruption.

Frequently Asked Questions

What happens when I set the model to "auto" in FreeLLMAPI?

When you specify model: "auto", the router initiates the fallback chain walk described around line 724 of router.ts. It expands the automatic selection into all available providers permitted by your API key's quota, then scores and ranks them to determine the optimal handler for your specific request.

How does FreeLLMAPI prevent selecting providers that have hit rate limits?

The router consults server/src/services/ratelimit.ts (specifically line 185) to check RPD limits before scoring. If your key has exhausted a provider's quota, the system benchmarks that provider for 24 hours, effectively removing it from the candidate pool until the limit resets.

Can I customize which providers FreeLLMAPI prioritizes for my account?

Yes, administrators can adjust routing behavior by calling the declarative configuration endpoints. The setCustomWeights function in server/src/services/declarative-config.ts allows you to apply custom weight multipliers to specific providers, while setRoutingStrategy switches between tier-first and rank-as-tiebreaker modes.

What occurs if the selected provider returns a 429 error?

The router implements transparent fallback logic. If the primary provider returns a 429 or other hard error, the system automatically retries the request with the next highest-scoring candidate from the fallback chain. This logic is implemented around line 735 in server/src/services/router.ts, ensuring high availability without client-side intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}