Understanding the In-Flight Lease Mechanism in FreeLLMAPI

The in-flight lease mechanism in FreeLLMAPI is a memory-only concurrency control system that prevents race conditions by tracking active API requests in a per-process Map, ensuring quota checks account for pending calls before dispatching new ones.

FreeLLMAPI uses an in-flight lease mechanism to solve the classic "check-then-act" race condition that emerges when multiple concurrent requests target the same API key. This lightweight bookkeeping layer, implemented in the tashfeenahmed/freellmapi repository, ensures that rate limit counters reflect both persisted usage and requests currently awaiting provider responses. By maintaining temporary leases in memory, the router can make accurate dispatch decisions without overloading keys, particularly on free-tier accounts with strict concurrency caps.

Why FreeLLMAPI Needs In-Flight Leases

The router selects API keys based on current usage counters—such as requests per minute or tokens per day—stored in the database. However, these counters are only updated after a provider call succeeds. Without an intermediate tracking layer, the following race condition occurs:

1. Request A checks quotas and passes all gates
2. Request B checks quotas before Request A completes
3. Both requests see the same low usage count and select the same key
4. The combined load exceeds the actual quota, triggering a flood of 429 errors

The in-flight lease mechanism closes this window by creating a temporary reservation that makes pending usage visible to subsequent quota checks.

How the In-Flight Lease Mechanism Works

Lease Acquisition in ratelimit.ts

When a key passes all gating checks, the router invokes acquireLease() defined in server/src/services/ratelimit.ts (lines 26-33). This function creates a record containing the platform, model ID, key ID, estimated token count, and creation timestamp, storing it in a global in-memory Map:

// ratelimit.ts – creates a lease and stores it in the global Map
export function acquireLease(
  platform: string,
  modelId: string,
  keyId: number,
  tokens: number,
  now = Date.now(),
): number {
  pruneLeases(now);                     // remove stale entries
  const id = nextLeaseId++;
  leases.set(id, { platform, modelId, keyId, tokens, createdAt: now });
  return id;
}

Source: [ratelimit.ts line 26-34](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts#L26-L34)

Attaching Leases to Routes

The router attaches the returned lease ID to the selected route object. In server/src/services/router.ts (lines 64-66), the lease becomes part of the route metadata that travels with the request through the fallback loop:

// router.ts – after all gating checks have passed
const leaseId = acquireLease(entry.platform, entry.model_id, key.id, estimatedTokens);
// The route object now carries `leaseId` and will release it once the request finishes

Source: [router.ts line 64-66](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts#L64-L66)

Provisional Usage Accounting

All quota-checking functions count both recorded database usage and active leases. The provisionalRequests() function in ratelimit.ts (lines 63-68) iterates the lease Map to calculate pending usage for specific key-model combinations:

function provisionalRequests(platform: string, modelId: string, keyId: number, now: number): number {
  pruneLeases(now);
  let count = 0;
  for (const lease of leases.values()) {
    if (lease.platform === platform && lease.modelId === modelId && lease.keyId === keyId) count++;
  }
  return count;
}

Source: [ratelimit.ts line 63-68](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts#L63-L68)

This ensures that canUseKeyConcurrency and provider-wide limits (requests per minute/day, token caps) see the true in-flight state before approving new dispatches.

Lease Lifecycle and Cleanup

Automatic Expiration via LEASE_MAX_AGE_MS

Leases exist only in memory and are never persisted to disk, ensuring a clean slate after process restarts. To prevent memory leaks from unexpected terminations, each lease carries a maximum age defined by LEASE_MAX_AGE_MS (line 76 of ratelimit.ts). The pruneLeases() function removes any entries older than this threshold during every acquisition and counting operation.

Explicit Release in the Fallback Loop

The normal cleanup path occurs in the finally block of the fallback loop in server/src/lib/fallback-loop.ts. The router explicitly calls releaseLease(leaseId) to remove the entry from the Map:

// Called in the finally block of the fallback loop
export function releaseLease(leaseId: number): void {
  leases.delete(leaseId);               // idempotent – safe if called twice
}

Source: [ratelimit.ts line 40-44](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts#L40-L44)

The release operation is idempotent; calling it twice for the same ID is harmless because the Map deletion silently fails if the key no longer exists.

Integration with Rate Limiting Gates

The in-flight lease mechanism backs the per-key concurrency cap (canUseKeyConcurrency) and feeds provisional usage into all provider-wide limits. By counting active leases together with persisted usage, the system guarantees that the router’s decision reflects the true occupancy state of each key. This prevents over-dispatching on free-tier accounts that are especially sensitive to concurrency limits, effectively converting optimistic request selection into a race-free operation.

Summary

• The in-flight lease mechanism solves check-then-act race conditions by tracking pending requests in a per-process Map before they complete.
• Leases are created via acquireLease() in ratelimit.ts and attached to routes in router.ts, making provisional usage visible to quota checks.
• Each lease automatically expires after LEASE_MAX_AGE_MS to prevent memory leaks, and is explicitly released in the finally block of fallback-loop.ts.
• The releaseLease() function is idempotent, ensuring safe double-release scenarios.
• This memory-only approach keeps accounting accurate under heavy parallel load without database write overhead for transient states.

Frequently Asked Questions

What is an in-flight lease in FreeLLMAPI?

An in-flight lease is a temporary memory reservation created when the router selects an API key for a request. Stored in a global Map in ratelimit.ts, it tracks the platform, model, key ID, and estimated tokens until the request completes, allowing subsequent quota checks to see the pending usage.

How does FreeLLMAPI prevent lease memory leaks?

The system uses two safeguards: a finally block in fallback-loop.ts that explicitly releases leases after requests finish, and an automatic expiration mechanism (LEASE_MAX_AGE_MS in ratelimit.ts) that prunes stale entries during every lease operation. Since leases are never persisted, process restarts also clear any orphaned entries.

Are in-flight leases shared across multiple FreeLLMAPI processes?

No. Leases are stored in a per-process Map and exist only in memory. They are not shared via database or distributed cache, meaning each server instance maintains its own view of in-flight requests. This design assumes sticky routing or accepts that different processes may have slightly divergent views of concurrency.

How does the lease mechanism affect rate limit calculations?

Quota-checking functions like provisionalRequests() and provisionalTokens() sum both recorded database usage and active leases before returning a count. This ensures that concurrency caps and token limits account for requests that have passed the router but not yet returned from the provider, preventing quota overruns.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}