How to Implement a Custom Provider Adapter for a Non-OpenAI API in FreeLLMAPI

To implement a custom provider adapter in FreeLLMAPI, extend the BaseProvider abstract class, implement the three required methods (chatCompletion, streamChatCompletion, and validateKey), translate the upstream API responses into OpenAI-compatible formats, and register the provider in server/src/providers/index.ts.

FreeLLMAPI (from the tashfeenahmed/freellmapi repository) unifies disparate LLM backends behind a single provider interface. By implementing a custom adapter, you can integrate any RESTful LLM service—even those with unique request formats or authentication schemes—into the proxy without modifying the core routing or quota-tracking logic.

Understanding the BaseProvider Contract

The foundation of every provider in tashfeenahmed/freellmapi is the BaseProvider abstract class defined in server/src/providers/base.ts. This contract enforces three mandatory methods:

• chatCompletion: Handles synchronous, non-streaming requests.
• streamChatCompletion: Handles server-sent event (SSE) streaming responses.
• validateKey: Verifies API key validity against the upstream service.

BaseProvider also provides utility methods for HTTP handling: fetchWithTimeout for aborted requests, makeId for generating unique identifiers, and readSseStream for parsing SSE streams. Reusing these utilities ensures consistent timeout behavior and error handling across all providers.

Step-by-Step Implementation

Create a Subclass of BaseProvider

Begin by creating a new TypeScript file in server/src/providers/. Your class must extend BaseProvider and define the platform identifier (used for routing and quota tracking) and a human-readable name.

import { BaseProvider, providerHttpError } from './base.js';
import type { ChatMessage, ChatCompletionResponse, ChatCompletionChunk, CompletionOptions } from '@freellmapi/shared/types.js';
import type { QuotaObservationContext } from '../services/provider-quota.js';

export class MyCoolProvider extends BaseProvider {
  readonly platform = 'mycool' as const;
  readonly name = 'My Cool Service';
  private readonly baseUrl: string;
  private readonly extraHeaders: Record<string, string>;

  constructor(opts: { baseUrl: string; extraHeaders?: Record<string, string> }) {
    super();
    this.baseUrl = opts.baseUrl;
    this.extraHeaders = opts.extraHeaders ?? {};
  }
  
  // Required methods implemented below...
}

Translate Request and Response Formats

The core responsibility of a custom provider adapter is protocol translation. In chatCompletion, construct the payload expected by your upstream API, then map the response into the OpenAI-compatible ChatCompletionResponse shape.

async chatCompletion(
  apiKey: string,
  messages: ChatMessage[],
  modelId: string,
  options?: CompletionOptions,
  quotaContext?: QuotaObservationContext,
): Promise<ChatCompletionResponse> {
  // Build provider-specific payload
  const payload = {
    model: modelId,
    history: messages,
    temperature: options?.temperature,
    maxTokens: options?.max_tokens,
  };

  // Execute with timeout handling
  const res = await this.fetchWithTimeout(`${this.baseUrl}/chat`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      Authorization: `Bearer ${apiKey}`,
      ...this.extraHeaders,
    },
    body: JSON.stringify(payload),
  });

  // Handle quota observation (optional but recommended)
  recordQuotaObservationsFromResponse(res, {
    platform: this.platform,
    keyId: quotaContext?.keyId,
    modelId,
    quotaPoolKey: quotaContext?.quotaPoolKey,
    endpoint: 'chat/completions',
  });

  // Error handling
  if (!res.ok) {
    const err = await res.json().catch(() => ({}));
    throw providerHttpError(res, `${this.name} API error ${res.status}: ${err.error?.message ?? res.statusText}`);
  }

  // Map upstream response to OpenAI format
  const upstream = await res.json();
  return {
    id: this.makeId(),
    object: 'chat.completion',
    created: Math.floor(Date.now() / 1000),
    model: modelId,
    choices: [{
      index: 0,
      message: {
        role: 'assistant',
        content: upstream.replyText, // Map field names appropriately
      },
      finish_reason: 'stop',
    }],
    usage: {
      prompt_tokens: upstream.usage?.promptTokens ?? 0,
      completion_tokens: upstream.usage?.completionTokens ?? 0,
      total_tokens: upstream.usage?.totalTokens ?? 0,
    },
  };
}

Handle Streaming Responses

For streaming support, implement streamChatCompletion as an async generator yielding ChatCompletionChunk objects. If your upstream API uses SSE, delegate to this.readSseStream() from the base class. For non-standard streaming protocols (like Google's Gemini), implement custom parsing logic similar to the pattern found in server/src/providers/google.ts.

async *streamChatCompletion(
  apiKey: string,
  messages: ChatMessage[],
  modelId: string,
  options?: CompletionOptions,
): AsyncGenerator<ChatCompletionChunk> {
  const res = await this.fetchWithTimeout(`${this.baseUrl}/chat/stream`, {
    method: 'POST',
    headers: {
      Authorization: `Bearer ${apiKey}`,
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ model: modelId, messages }),
  });

  if (!res.ok) {
    throw providerHttpError(res, `${this.name} streaming error ${res.status}`);
  }

  // Re-use generic SSE parser if compatible
  yield* this.readSseStream(res);
}

Implement Key Validation

The validateKey method performs a lightweight health check. Return true for valid credentials, false for invalid ones. Most implementations issue a GET request to a /models or /status endpoint and check for 401/403 status codes.

async validateKey(apiKey: string): Promise<boolean> {
  const res = await this.fetchWithTimeout(`${this.baseUrl}/status`, {
    method: 'GET',
    headers: { Authorization: `Bearer ${apiKey}` },
  });
  return res.status !== 401 && res.status !== 403;
}

Registering Your Custom Provider

After implementation, expose the provider to the router by registering it in server/src/providers/index.ts:

import { MyCoolProvider } from './my-cool-provider.js';

register(new MyCoolProvider({
  baseUrl: 'https://api.mycoolservice.com/v1',
  extraHeaders: { 'X-Custom-Header': 'value' },
}));

For dynamic, user-supplied endpoints, the repository already contains a resolver pattern. When platform === 'custom', the code constructs a fresh OpenAICompatProvider on-the-fly. Replace this with your custom class if the endpoint is not OpenAI-compatible:

if (platform === 'custom' && baseUrl) {
  return new MyCoolProvider({ baseUrl: baseUrl.trim() });
}

Handling Keyless Authentication

Some services offer anonymous tiers or use IP-based authentication. To support this, set this.keyless = true in your constructor. When this flag is enabled, the proxy layer omits the Authorization header from requests to the upstream service.

constructor(opts: { baseUrl: string }) {
  super();
  this.baseUrl = opts.baseUrl;
  this.keyless = true; // No API key required
}

Complete Code Example

Here is a minimal, production-ready provider file template:

// server/src/providers/my-cool-provider.ts
import type { ChatMessage, ChatCompletionResponse, ChatCompletionChunk, CompletionOptions } from '@freellmapi/shared/types.js';
import { BaseProvider, providerHttpError } from './base.js';
import { recordQuotaObservationsFromResponse, type QuotaObservationContext } from '../services/provider-quota.js';

export class MyCoolProvider extends BaseProvider {
  readonly platform = 'mycool' as const;
  readonly name = 'My Cool Service';
  private readonly baseUrl: string;

  constructor(opts: { baseUrl: string }) {
    super();
    this.baseUrl = opts.baseUrl;
  }

  async chatCompletion(
    apiKey: string,
    messages: ChatMessage[],
    modelId: string,
    options?: CompletionOptions,
    quotaContext?: QuotaObservationContext,
  ): Promise<ChatCompletionResponse> {
    const payload = { model: modelId, messages, temperature: options?.temperature };
    const res = await this.fetchWithTimeout(`${this.baseUrl}/chat`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json', Authorization: `Bearer ${apiKey}` },
      body: JSON.stringify(payload),
    });

    if (!res.ok) throw providerHttpError(res, `${this.name} error`);
    
    const data = await res.json();
    return {
      id: this.makeId(),
      object: 'chat.completion',
      created: Math.floor(Date.now() / 1000),
      model: modelId,
      choices: [{ index: 0, message: { role: 'assistant', content: data.text }, finish_reason: 'stop' }],
      usage: { prompt_tokens: 0, completion_tokens: 0, total_tokens: 0 },
    };
  }

  async *streamChatCompletion(apiKey: string, messages: ChatMessage[], modelId: string): AsyncGenerator<ChatCompletionChunk> {
    const res = await this.fetchWithTimeout(`${this.baseUrl}/stream`, { /* ... */ });
    yield* this.readSseStream(res);
  }

  async validateKey(apiKey: string): Promise<boolean> {
    const res = await this.fetchWithTimeout(`${this.baseUrl}/status`, { headers: { Authorization: `Bearer ${apiKey}` } });
    return res.ok;
  }
}

Summary

• Extend BaseProvider from server/src/providers/base.ts to create a new adapter.
• Implement three abstract methods: chatCompletion, streamChatCompletion, and validateKey.
• Translate protocols by mapping upstream request/response formats to OpenAI-compatible types defined in shared/types.ts.
• Reuse base utilities like fetchWithTimeout and readSseStream for consistent HTTP and SSE handling.
• Register the provider in server/src/providers/index.ts using the register() function.
• Set this.keyless = true for services that do not require API key authentication.

Frequently Asked Questions

What is the minimum number of methods I must implement in a custom provider?

You must implement three abstract methods: chatCompletion for synchronous responses, streamChatCompletion for streaming (can return an empty generator if unsupported), and validateKey for credential verification. The base class provides HTTP utilities like fetchWithTimeout and readSseStream to simplify these implementations.

How do I handle APIs that return formats different from OpenAI?

Map the upstream response fields to the OpenAI-compatible ChatCompletionResponse structure in your chatCompletion method. For complex transformations—such as Google's Gemini multimodal responses—refer to server/src/providers/google.ts as a reference for translating nested or differently named fields into the standard choices, message, and usage schema.

Can I support both custom static providers and dynamic user-provided endpoints?

Yes. Register static providers with register(new YourProvider()) in server/src/providers/index.ts. For dynamic endpoints, modify the resolver logic in the same file where platform === 'custom' to instantiate your provider class with a user-supplied baseUrl, similar to how OpenAICompatProvider is constructed on-the-fly.

Where should I place quota tracking logic for my custom provider?

Call recordQuotaObservationsFromResponse() within your chatCompletion method after receiving the upstream response, as shown in the code skeleton. Pass the res object along with platform, keyId, and modelId from the QuotaObservationContext to enable automatic quota tracking and rate-limiting without manual intervention.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}