How to Add a New Provider to FreeLLMAPI: A Step-by-Step Implementation Guide

To add a new provider to FreeLLMAPI, create a class extending BaseProvider (or reuse OpenAICompatProvider for OpenAI-compatible APIs), implement the three required methods (chatCompletion, streamChatCompletion, and validateKey), and register the instance in server/src/providers/index.ts using the register() function.

FreeLLMAPI routes every LLM request through a centralized provider registry that maps platform identifiers to provider implementations. Adding a new provider to the tashfeenahmed/freellmapi repository requires implementing the abstract contract defined in the base class and exposing your implementation to the router. This guide covers both the quick path for OpenAI-compatible endpoints and the full custom implementation for proprietary APIs.

Understanding the Provider Architecture

FreeLLMAPI uses a registry pattern centered on server/src/providers/index.ts. This file maintains a singleton providers map that connects platform strings (like "openai" or "anthropic") to provider instances.

Each provider must implement the BaseProvider abstract class from server/src/providers/base.ts. The router in server/src/services/router.ts retrieves the appropriate provider using getProvider(platform) and delegates requests to three core methods:

• chatCompletion – Handles synchronous completion requests
• streamChatCompletion – Handles Server-Sent Events (SSE) streaming responses
• validateKey – Validates API keys before routing requests

The BaseProvider class supplies essential utilities including fetchWithTimeout, readSseStream, and validationResult, so you only need to implement the HTTP call logic specific to your endpoint.

Method 1: Adding an OpenAI-Compatible Provider

If your target API follows the OpenAI request/response format, use the existing OpenAICompatProvider. This generic implementation already handles the three abstract methods; you only need to configure the base URL and metadata.

Create a new file or add to an existing provider file:

// server/src/providers/myai.ts
import { OpenAICompatProvider } from './openai-compat.js';
import { register } from './index.js';

register(new OpenAICompatProvider({
  platform: 'myai',
  name: 'MyAI',
  baseUrl: 'https://api.myai.com/v1',
  timeoutMs: 90_000,           // Optional: increase timeout for cold starts
  extraHeaders: { 'X-Custom-Header': 'free-llmapi' }
}));

This approach works for most modern LLM APIs that mimic the OpenAI /v1/chat/completions endpoint structure.

Method 2: Creating a Custom Provider

For APIs with non-standard formats, extend BaseProvider directly in server/src/providers/myai.ts:

import { BaseProvider, providerHttpError } from './base.js';
import type { 
  ChatMessage, 
  ChatCompletionResponse, 
  CompletionOptions 
} from '@freellmapi/shared/types.js';

export class MyAIProvider extends BaseProvider {
  readonly platform = 'myai' as const;
  readonly name = 'MyAI';

  async chatCompletion(
    apiKey: string, 
    messages: ChatMessage[], 
    modelId: string, 
    options?: CompletionOptions
  ): Promise<ChatCompletionResponse> {
    const res = await this.fetchWithTimeout(
      'https://api.myai.com/v1/chat/completions',
      {
        method: 'POST',
        headers: { 
          'Authorization': `Bearer ${apiKey}`, 
          'Content-Type': 'application/json' 
        },
        body: JSON.stringify({ model: modelId, messages, ...options }),
      }
    );
    
    if (!res.ok) {
      throw providerHttpError(res, 'MyAI chat failed', await res.json());
    }
    
    return res.json() as Promise<ChatCompletionResponse>;
  }

  async *streamChatCompletion(
    apiKey: string, 
    messages: ChatMessage[], 
    modelId: string, 
    options?: CompletionOptions
  ) {
    const res = await this.fetchWithTimeout(
      'https://api.myai.com/v1/chat/completions',
      {
        method: 'POST',
        headers: { 
          'Authorization': `Bearer ${apiKey}`, 
          'Content-Type': 'application/json' 
        },
        body: JSON.stringify({ 
          model: modelId, 
          messages, 
          stream: true, 
          ...options 
        }),
      }
    );
    
    if (!res.ok) {
      throw providerHttpError(res, 'MyAI stream failed', await res.json());
    }
    
    // Re-use the SSE helper from BaseProvider
    yield* this.readSseStream(res);
  }

  async validateKey(apiKey: string): Promise<boolean> {
    const res = await this.fetchWithTimeout(
      'https://api.myai.com/v1/models',
      { headers: { Authorization: `Bearer ${apiKey}` } }
    );
    return this.validationResult(res);
  }
}

The fetchWithTimeout method handles request timeouts and retries. The readSseStream method parses Server-Sent Events into the standard chunk format. Use providerHttpError to standardize error responses across the application.

Registering Your Provider

After creating your provider class, register it in server/src/providers/index.ts:

// server/src/providers/index.ts
import { MyAIProvider } from './myai.js';

// ... existing registrations ...
register(new MyAIProvider());

The register function inserts the instance into the internal providers map, making it available to getProvider() calls throughout the application. The platform property you defined in your class serves as the lookup key.

Configuration and Customization

Pass an options object to the constructor to customize behavior:

register(new MyAIProvider({ 
  timeoutMs: 120_000,  // Default is typically 60s
  extraHeaders: { 'X-Provider-Source': 'freellmapi' }
}));

Available configuration options include:

• timeoutMs – Request timeout in milliseconds
• extraHeaders – Headers appended to every request
• retryAttempts – Number of automatic retries for failed requests

These options are processed by the BaseProvider constructor and applied automatically to all HTTP requests made by your provider.

Testing Your Implementation

Validate your provider integration using the existing test suite:

pnpm test

# or

npm test

Add unit tests under server/src/services/__tests__/ to verify edge cases specific to your provider. Ensure your validateKey method correctly handles both valid and invalid API keys, and verify that streaming responses properly yield chunks in the expected format.

Summary

• Extend BaseProvider or reuse OpenAICompatProvider depending on API compatibility
• Implement three methods: chatCompletion, streamChatCompletion, and validateKey
• Register the instance in server/src/providers/index.ts using the register() function
• Configure timeouts and headers via the constructor options object
• Use built-in helpers like fetchWithTimeout and readSseStream to handle networking complexity
• The router automatically discovers your provider via the platform identifier without requiring modifications to router.ts

Frequently Asked Questions

Do I need to modify the router to add a new provider?

No. The router in server/src/services/router.ts dynamically retrieves providers using getProvider(platform). As long as you register your provider in server/src/providers/index.ts, the router automatically discovers it without code changes.

Can I add a provider that only supports non-streaming responses?

Yes. Implement chatCompletion normally for synchronous responses. For streamChatCompletion, you can either throw a Method not implemented error or aggregate the full response into a single chunk. The base class handles both patterns gracefully.

How do I handle API rate limits and retries?

The BaseProvider class automatically parses Retry-After headers and implements exponential backoff. You can customize retry behavior by passing retryAttempts in the constructor options. For provider-specific rate limit handling, override the handleError method in your subclass.

What is the difference between platform and name properties?

The platform property is a unique string identifier used internally for routing (e.g., "myai"), while the name property is a human-readable display name shown in logs and user interfaces (e.g., "MyAI Enterprise"). The platform must be unique across all registered providers.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}