How Fallback Chain Configuration Works in the FreeLLMAPI Dashboard

The fallback chain configuration in FreeLLMAPI is managed through a SQLite-backed Express API that serves an ordered list of models with dynamic penalties, allowing admins to reorder, enable/disable, and apply routing presets via REST endpoints consumed by a React dashboard.

The tashfeenahmed/freellmapi repository provides an open-source LLM routing platform that intelligently distributes requests across multiple providers. At the heart of this system lies the fallback chain configuration, which defines the ordered sequence of models the platform attempts when primary requests fail or require rerouting.

SQLite Data Model and the Primary Endpoint

The fallback chain persists in the fallback_config table, which stores static priority rankings and enabled states for each model. In server/src/routes/fallback.ts (lines 58‑71), the GET /api/fallback endpoint executes a SQL join between fallback_config and the models table to construct a comprehensive response:

SELECT fc.model_db_id, fc.priority, fc.enabled,
       m.platform, m.model_id, m.display_name, …
FROM fallback_config fc
JOIN models m ON m.id = fc.model_db_id
WHERE m.enabled = 1
ORDER BY fc.priority ASC

The response enriches raw database fields with runtime metrics. Each entry includes:

• priority – The static order set by the administrator
• effectivePriority – Calculated as priority + penalty (lines 99‑105)
• penalty and rateLimitHits – Sourced from getAllPenalties() in server/src/services/router.ts
• keyCount – Number of enabled API keys for the model’s platform (lines 73‑80)
• monthlyTokenBudgetTokens – Parsed from budget strings via parseBudget() in server/src/lib/budget.ts (lines 122‑125)
• groupKey, canonicalId, groupLabel – Logical grouping metadata from getModelGroups() (lines 88‑94)

Because the server supplies all derived fields in a single JSON payload, the client renders the table without additional lookups.

Updating and Reordering the Chain

Administrators modify the fallback chain configuration through two primary mechanisms.

Full Chain Replacement

The PUT /api/fallback endpoint accepts a complete array of objects containing modelDbId, priority, and enabled boolean. The server executes a database transaction that updates each corresponding row in fallback_config, ensuring atomic consistency when reordering or toggling model availability.

// Send the whole edited chain back to the server
await apiFetch('/api/fallback', {
  method: 'PUT',
  body: JSON.stringify(editedEntries), // [{modelDbId, priority, enabled}, …]
});

Preset-Based Sorting

For automated optimization, POST /api/fallback/sort/:preset (lines 99‑126) applies algorithmic sorting based on strategic criteria:

• intelligence – Sorts by model size-tier then intelligence_rank
• speed – Orders by speed_rank
• budget – Calculates a budget score from monthly_token_budget or tpd_limit using getBudgetScore (lines 76‑96)

// server/src/routes/fallback.ts
fallbackRouter.post('/sort/:preset', (req, res) => {
  const preset = String(req.params.preset);
  // … compute `models` array based on preset …
  const update = db.prepare('UPDATE fallback_config SET priority = ? WHERE model_db_id = ?');
  db.transaction(() => models.forEach((m, i) => update.run(i + 1, m.id)))();
  res.json({ success: true, preset });
});

Routing Strategy Configuration

Beyond static ordering, the dashboard manages dynamic routing behavior through the bandit strategy system. The GET /api/fallback/routing endpoint (lines 18‑22) returns the active strategy, preset weights, and per-model scores.

Administrators switch strategies via PUT /api/fallback/routing (lines 24‑56), selecting from priority, balanced, smartest, fastest, reliable, or custom. When selecting custom, the system persists a weight vector (reliability, speed, intelligence) through setCustomWeights.

await apiFetch('/api/fallback/routing', {
  method: 'PUT',
  body: JSON.stringify({ 
    strategy: 'custom', 
    weights: { reliability: 0.5, speed: 0.3, intelligence: 0.2 } 
  }),
});

The preset weights and scoring logic reside in server/src/services/scoring.ts, which exports BANDIT_PRESETS defining how each strategy balances competing objectives.

Token Budget Visualization

The dashboard renders token consumption through GET /api/fallback/token-usage (lines 28‑107). This endpoint aggregates:

• totalBudget – Combined token budget across all models
• totalUsed – Tokens consumed during the current month
• Per-model breakdowns including limits and enabled flags

When an active profile exists, the endpoint intelligently pulls chain data from profile_models rather than the global fallback_config.

React Frontend Integration

The client implementation in client/src/pages/FallbackPage.tsx consumes these endpoints using the apiFetch helper with React Query’s useQuery(['fallback']) for caching.

// client/src/pages/FallbackPage.tsx
const { data: fallbackEntries = [] } = useQuery<FallbackEntry[]>({
  queryKey: ['fallback'],
  queryFn: () => apiFetch('/api/fallback'),
});

The UI renders each model’s display name, static priority, effective priority, and fallback attempt badges from the X-Fallback-Attempts response header. Drag-and-drop reordering and enable toggles trigger the PUT /api/fallback endpoint, while routing strategy changes invoke PUT /api/fallback/routing. Because server/src/routes/fallback.ts pre-calculates all business logic, the client remains a thin presentation layer.

Summary

• The fallback chain configuration resides in the SQLite fallback_config table, joined with models to provide rich metadata
• The GET /api/fallback endpoint supplies static priorities, dynamic penalties from getAllPenalties(), and budget calculations from parseBudget()
• Chain updates occur atomically via PUT /api/fallback, while POST /api/fallback/sort/:preset enables algorithmic reordering by intelligence, speed, or budget
• Routing strategies (priority, balanced, smartest, fastest, reliable, custom) control how penalties affect request distribution
• The React dashboard in FallbackPage.tsx consumes these endpoints through react-query, displaying effectivePriority and token usage without client-side business logic

Frequently Asked Questions

Where is the fallback chain data physically stored?

The configuration persists in a SQLite table named fallback_config, which contains model_db_id, priority, and enabled columns. This table joins against the models table in server/src/routes/fallback.ts to enrich entries with platform details, capabilities, and display names.

How does the dashboard calculate effectivePriority?

The effectivePriority field equals the static priority value plus a dynamic penalty score. The penalty derives from getAllPenalties() in server/src/services/router.ts, which factors in rateLimitHits and other runtime metrics to deprioritize struggling models automatically.

Can I automate the ordering based on model performance or cost?

Yes. The POST /api/fallback/sort/:preset endpoint supports three automated presets: intelligence (ranks by capability), speed (ranks by latency), and budget (ranks by token cost). Each preset recalculates priorities and updates the fallback_config table atomically.

What happens when I switch routing strategies?

When you PUT to /api/fallback/routing with a new strategy, the system updates how penalties are applied to the fallback chain. Strategies like smartest or fastest use predefined weights from BANDIT_PRESETS in server/src/services/scoring.ts, while custom allows you to specify exact weights for reliability, speed, and intelligence via setCustomWeights.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}