FreeLLMAPI Provider Adapter Architecture: How to Add a New LLM Provider

FreeLLMAPI normalizes access to dozens of LLM backends through an abstract BaseProvider class that enforces a three-method contract (chatCompletion, streamChatCompletion, validateKey), coupled with a singleton registry pattern that allows new providers to be integrated by extending the base class and registering the instance without modifying the router or business logic.

FreeLLMAPI (tashfeenahmed/freellmapi) unifies disparate LLM APIs behind a single interface. The provider adapter architecture decouples vendor-specific wire protocols from core routing logic, enabling developers to add support for new platforms by implementing a predictable TypeScript interface.

Provider Adapter Architecture Overview

The BaseProvider Abstract Class

The foundation of the system is BaseProvider, defined in server/src/providers/base.ts. This abstract class declares three methods that every concrete provider must implement: chatCompletion, streamChatCompletion, and validateKey. It also supplies shared utilities including fetchWithTimeout for HTTP requests with global proxy support, readSseStream for parsing Server-Sent Events, and providerHttpError for standardized error handling.

Concrete Provider Implementations

Each supported backend extends BaseProvider in its own file under server/src/providers/. For example, GoogleProvider handles the Gemini API in server/src/providers/google.ts, while OpenAICompatProvider in server/src/providers/openai-compat.ts serves as a generic adapter for any OpenAI-compatible service. These classes translate the universal OpenAI-style schema into vendor-specific request formats.

The Provider Registry

The registry in server/src/providers/index.ts maintains a singleton Map<Platform, BaseProvider> that holds all built-in providers. Helper functions getProvider, resolveProvider, hasProvider, and getAllProviders expose this map to the rest of the application. Registration happens via register(new ProviderClass()), while the special custom platform is resolved dynamically from user-supplied URLs.

Request Flow

When a request arrives, the router (server/src/services/router.ts) extracts the desired platform identifier and calls getProvider(platform). The returned instance executes the requested operation using proxyFetch for consistent tracing and proxy configuration. Because all providers adhere to the same abstract interface, higher-level services such as quota tracking and health checks remain agnostic to backend implementation details.

How to Add a New Provider to FreeLLMAPI

Adding support for a new LLM backend requires five discrete steps. You will create a new TypeScript file, implement three abstract methods, register the class, and update the platform type union.

Step 1: Create the Provider File

Create a new file under server/src/providers/ (e.g., mycloud.ts). Import the base class and types:

import { BaseProvider, providerHttpError } from './base.js';
import type { ChatMessage, ChatCompletionResponse, ChatCompletionChunk, QuotaObservationContext } from '@freellmapi/shared/types.js';
import { recordQuotaObservationsFromResponse } from '../services/provider-quota.js';

Step 2: Extend BaseProvider

Define a class that extends BaseProvider. Declare the platform identifier and name as readonly constants. Optionally set keyless = true if the service requires no authentication.

export class MyCloudProvider extends BaseProvider {
  readonly platform = 'mycloud' as const;
  readonly name = 'MyCloud AI';
  // keyless = false; // set to true if no API key required
}

Step 3: Implement the Required Methods

Implement the three abstract methods: chatCompletion, streamChatCompletion, and validateKey.

chatCompletion sends non-streaming requests:

async chatCompletion(
  apiKey: string,
  messages: ChatMessage[],
  modelId: string,
  options?: CompletionOptions,
  quotaContext?: QuotaObservationContext,
): Promise<ChatCompletionResponse> {
  const res = await this.fetchWithTimeout('https://api.mycloud.ai/v1/chat/completions', {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ model: modelId, messages, ...options }),
  });

  recordQuotaObservationsFromResponse(res, {
    platform: this.platform,
    keyId: quotaContext?.keyId,
    modelId,
    quotaPoolKey: quotaContext?.quotaPoolKey,
    endpoint: 'chat/completions',
  });

  if (!res.ok) {
    const errBody = await res.json().catch(() => ({}));
    throw providerHttpError(res, `MyCloud error ${res.status}: ${errBody?.error?.message ?? res.statusText}`);
  }

  const data = await res.json() as ChatCompletionResponse;
  data._routed_via = { platform: this.platform, model: modelId };
  return data;
}

streamChatCompletion yields SSE chunks using the base class helper:

async *streamChatCompletion(
  apiKey: string,
  messages: ChatMessage[],
  modelId: string,
  options?: CompletionOptions,
  quotaContext?: QuotaObservationContext,
): AsyncGenerator<ChatCompletionChunk> {
  const res = await this.fetchWithTimeout('https://api.mycloud.ai/v1/chat/completions', {
    method: 'POST',
    headers: { Authorization: `Bearer ${apiKey}`, 'Content-Type': 'application/json' },
    body: JSON.stringify({ model: modelId, messages, stream: true, ...options }),
  });

  recordQuotaObservationsFromResponse(res, {
    platform: this.platform,
    keyId: quotaContext?.keyId,
    modelId,
    quotaPoolKey: quotaContext?.quotaPoolKey,
    endpoint: 'chat/completions',
  });
  
  if (!res.ok) {
    const err = await res.json().catch(() => ({}));
    throw providerHttpError(res, `MyCloud streaming error ${res.status}: ${err?.error?.message ?? res.statusText}`);
  }
  
  yield* this.readSseStream(res);
}

validateKey checks credential validity:

async validateKey(apiKey: string, quotaContext?: QuotaObservationContext): Promise<boolean> {
  const res = await this.fetchWithTimeout('https://api.mycloud.ai/v1/models', {
    method: 'GET',
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  // Return false only for confirmed bad credentials (401/403)
  return res.status !== 401 && res.status !== 403;
}

Step 4: Register the Provider

Import and register your class in server/src/providers/index.ts:

import { MyCloudProvider } from './mycloud.js';
register(new MyCloudProvider());

Once registered, the router automatically recognizes the new platform without code changes.

Step 5: Update Platform Types

If your platform uses a strict enum rather than string literals, add the identifier to the Platform union in shared/types.ts:

export type Platform = 'openai' | 'google' | 'mycloud' | 'custom';

Key Files in the Provider System

Understanding the codebase structure helps when debugging or extending adapters:

• server/src/providers/base.ts – Defines BaseProvider with abstract methods and shared utilities like fetchWithTimeout and readSseStream.
• server/src/providers/index.ts – Singleton registry exposing getProvider, register, and resolveProvider.
• server/src/providers/openai-compat.ts – Reference implementation for OpenAI-compatible APIs.
• server/src/providers/google.ts – Reference implementation for Gemini, showing non-OpenAI message translation.
• server/src/services/router.ts – Entry point that selects providers via the registry.
• shared/types.ts – Contains Platform type definitions and shared interfaces.

Summary

FreeLLMAPI's provider adapter system decouples backend specifics from core routing logic through an abstract class and registry pattern. Key takeaways include:

• Extend BaseProvider in server/src/providers/base.ts to implement chatCompletion, streamChatCompletion, and validateKey.
• Use shared helpers like fetchWithTimeout and readSseStream to ensure consistent timeout, proxy, and SSE handling.
• Register the instance in server/src/providers/index.ts using register(new YourProvider()).
• Update Platform types in shared/types.ts if using strict type unions.
• Leverage quota tracking by calling recordQuotaObservationsFromResponse in your HTTP handlers.

Frequently Asked Questions

Do I need to modify the router to add a new provider?

No. The router in server/src/services/router.ts dynamically resolves providers through the registry's getProvider function. Once you register your new class in server/src/providers/index.ts, the router automatically recognizes it without additional changes.

How does streaming work with the BaseProvider?

The BaseProvider class provides a readSseStream helper that parses Server-Sent Events. In your streamChatCompletion implementation, await the HTTP response, validate it, then yield this.readSseStream(res) to push chunks to the client while maintaining back-pressure handling.

Can I create a provider that doesn't require an API key?

Yes. Set the keyless property to true in your provider class. When keyless is enabled, the system skips validation checks and quota observations for that provider, treating it as a public or self-hosted endpoint.

What error handling pattern should I follow?

Use the providerHttpError utility from server/src/providers/base.ts to throw standardized errors. Return false from validateKey only for confirmed authentication failures (HTTP 401/403). For network-level failures, allow exceptions to propagate so health checks mark the key as "inconclusive" rather than definitively bad.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}