Default Fallback Time Budget in FreeLLMAPI: Configuration and Usage Guide

The default fallback time budget in FreeLLMAPI is 45 seconds (45,000 ms), defined as a hard-coded constant in the fallback loop implementation.

This setting controls how long the router keeps retrying failed provider calls before aborting further attempts. Understanding this limit is essential for tuning resilience in production deployments of tashfeenahmed/freellmapi.

Where the Default Is Defined

The default fallback time budget originates in server/src/lib/fallback-loop.ts. This file defines the core retry logic and exposes the constant FALLBACK_TIME_BUDGET_MS set to 45000.

The implementation enforces this budget as a wall‑clock ceiling. Once elapsed, the router stops spawning new provider attempts and returns an exhaustion error to the caller.

Configuration Methods

FreeLLMAPI provides two ways to override the default:

Environment Variable

Set FALLBACK_TIME_BUDGET_MS in your .env file or shell environment. The .env.example file documents this explicitly:

# Wall-clock budget for provider failover, in milliseconds (default: 45000)

FALLBACK_TIME_BUDGET_MS=45000

Runtime Settings API

The fallback_time_budget_ms setting key allows dynamic adjustment without restarting the server.

Code Examples

Check the Active Budget Programmatically

import { FALLBACK_TIME_BUDGET_MS } from './server/src/lib/fallback-loop';

const activeBudget = process.env.FALLBACK_TIME_BUDGET_MS 
  ? parseInt(process.env.FALLBACK_TIME_BUDGET_MS, 10) 
  : FALLBACK_TIME_BUDGET_MS;

console.log(`Fallback time budget: ${activeBudget} ms`);

Override via Environment Variable

# Extend budget to 90 seconds for slower providers

FALLBACK_TIME_BUDGET_MS=90000

Update at Runtime via API

await fetch('http://localhost:3001/api/settings', {
  method: 'PUT',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ 
    key: 'fallback_time_budget_ms', 
    value: '120000' 
  })
});

Key Source Files

Understanding the complete implementation requires examining these files in tashfeenahmed/freellmapi:

• server/src/lib/fallback-loop.ts — Defines FALLBACK_TIME_BUDGET_SETTING and the default 45000 value; implements budget enforcement logic.

• .env.example — Documents the environment variable and its default.

• server/src/__tests__/lib/fallback-loop.test.ts — Contains unit tests including budget manipulation (e.g., temporarily setting to 999 ms for fast-fail testing).

• server/src/services/router.ts — Consumes the budget to gate retry attempts against elapsed wall‑clock time.

How the Budget Enforces Limits

The fallback loop in fallback-loop.ts tracks elapsed time across provider attempts. When the cumulative duration exceeds the configured budget:

1. No new provider calls are initiated.
2. Pending in-flight requests may continue but no additional retries spawn.
3. The router returns an exhaustion error indicating fallback time budget exceeded.

This prevents cascading timeouts from stalling client requests indefinitely.

Summary

• Default fallback time budget: 45,000 ms (45 seconds) as defined in server/src/lib/fallback-loop.ts.
• Override options: FALLBACK_TIME_BUDGET_MS environment variable or fallback_time_budget_ms runtime setting.
• Purpose: Caps total wall‑clock time spent retrying failed provider calls.
• Enforcement point: server/src/services/router.ts consults the budget before each retry attempt.

Frequently Asked Questions

Can I disable the fallback time budget entirely?

No. FreeLLMAPI requires a positive budget to prevent unbounded retry loops. Setting FALLBACK_TIME_BUDGET_MS=0 or a negative value would likely trigger validation errors or immediate exhaustion, as implied by the test suite's strict positive assertions in fallback-loop.test.ts.

What happens if a single provider call exceeds the budget?

The budget applies to cumulative retry time, not individual call duration. An in-flight request that exceeds the remaining budget is not forcibly terminated; rather, no additional provider attempts start once the threshold is crossed.

How does the setting interact with per-provider timeouts?

These are independent controls. The fallback time budget governs total retry window across all providers, while individual providers may have their own request timeouts. Configure both to ensure slower providers don't consume disproportionate budget.

Is the default budget sufficient for most deployments?

The 45‑second default suits typical LLM provider latencies with 2–3 retry attempts. For providers with higher p99 latencies or more retry tiers, increase via FALLBACK_TIME_BUDGET_MS as demonstrated in the .env.example file.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}