How to Set Up Custom OpenAI-Compatible Endpoints with FreeLLMAPI

Send a POST request to /keys/custom with your baseUrl and model list, then use the registered model names in standard OpenAI API calls—FreeLLMAPI handles the rest automatically.

FreeLLMAPI is an open-source LLM gateway that unifies multiple providers behind a single OpenAI-compatible API. While it ships with built-in support for Groq, Mistral, and OpenRouter, its custom provider system lets you integrate any OpenAI-compatible endpoint—whether that's Ollama running locally, a self-hosted vLLM instance, or a third-party API service. This guide walks through the complete setup process based on the actual source implementation in tashfeenahmed/freellmapi.

---

Understanding the Custom Provider Architecture

FreeLLMAPI treats custom endpoints as dynamic providers that are created on-the-fly rather than initialized at startup. This differs from built-in providers like Groq or Mistral, which are instantiated once in server/src/providers/index.ts.

The architecture follows this flow:

1. Registration: Your POST /keys/custom request stores the endpoint configuration in the database
2. Model binding: Each model you specify gets inserted into the models table with a foreign key to your endpoint
3. Runtime resolution: When a chat request arrives, resolveProvider() constructs an OpenAICompatProvider instance using your stored baseUrl
4. Request forwarding: The provider forwards calls to baseUrl + '/v1/chat/completions' with your stored API key

This design means you don't need to restart the server or modify configuration files—endpoints are immediately available after registration.

---

Registering a Custom Endpoint

Required Parameters

The POST /keys/custom endpoint in server/src/routes/keys.ts expects:

Parameter	Type	Required	Description
baseUrl	string	Yes	The OpenAI-compatible base URL (e.g., http://localhost:11434/v1)
models	array	Yes	List of model objects or strings to register
apiKey	string	No	Authentication key, or omit/"no-key" for keyless services
label	string	No	Human-readable name for this endpoint

Example: Registering Ollama

curl -X POST https://your-llmapi.example.com/keys/custom \
  -H "Content-Type: application/json" \
  -d '{
    "baseUrl": "http://localhost:11434/v1",
    "apiKey": "no-key",
    "label": "Local Ollama",
    "models": [
      {"model": "llama3:8b", "displayName": "Llama 3 8B"},
      {"model": "phi2", "displayName": "Phi-2"}
    ]
  }'

Response structure:

{
  "success": true,
  "keyId": 23,
  "platform": "custom",
  "baseUrl": "http://localhost:11434/v1",
  "model": "llama3:8b",
  "displayName": "Llama 3 8B",
  "models": [
    {"modelDbId": 101, "model": "llama3:8b", "displayName": "Llama 3 8B"},
    {"modelDbId": 102, "model": "phi2", "displayName": "Phi-2"}
  ],
  "maskedKey": "no-key"
}

The keysRouter.post('/custom', ...) handler in server/src/routes/keys.ts performs three operations: it creates or updates a row in api_keys with platform = 'custom', normalizes the model list to handle both string and object formats, and inserts each model into the models table with the appropriate foreign key reference.

---

How Provider Resolution Works at Runtime

When a client sends a chat completion request using a model from your custom endpoint, FreeLLMAPI needs to determine where to route it. This happens through the resolveProvider() function in server/src/providers/index.ts:

export function resolveProvider(platform: Platform, baseUrl?: string | null): BaseProvider | undefined {
  if (platform === 'custom') {
    const trimmed = baseUrl?.trim();
    if (!trimmed) return undefined;
    return new OpenAICompatProvider({
      platform: 'custom',
      name: 'Custom (OpenAI-compatible)',
      baseUrl: trimmed,
      timeoutMs: CUSTOM_PROVIDER_TIMEOUT_MS, // 30 seconds
    });
  }
  return providers.get(platform);
}

Key details:

• Built-in providers are cached in a Map<Platform, BaseProvider>
• Custom providers are instantiated fresh for each request using the stored baseUrl
• Custom providers receive an extended 30-second timeout (CUSTOM_PROVIDER_TIMEOUT_MS) compared to built-in providers, accommodating slower self-hosted inference

The getProviderForModel() function handles the database lookup: it queries the models table by model identifier, retrieves the associated api_keys row including the base_url field, and calls resolveProvider() with the appropriate platform and URL.

---

The OpenAICompatProvider Implementation

The OpenAICompatProvider class in server/src/providers/openai-compat.ts provides the actual transport layer for custom endpoints. It extends BaseProvider and implements four key methods:

Authentication Handling

The authHeader() method supports both authenticated and keyless services:

private authHeader(apiKey: string): Record<string, string> {
  if (apiKey === 'no-key') {
    return {};
  }
  return { 'Authorization': `Bearer ${apiKey}` };
}

When you register with "no-key" or omit the apiKey field, the provider sends requests without an Authorization header—essential for local services like Ollama that don't require authentication.

Request Routing

The provider normalizes the base URL by stripping trailing /v1/ paths, then constructs full URLs:

• Chat completions: ${baseUrl}/v1/chat/completions
• Model listing: ${baseUrl}/v1/models (used for key validation)

Timeout Configuration

The constructor accepts a timeoutMs parameter (defaulting to 30 seconds) and uses AbortController to enforce it:

private async fetchWithTimeout(url: string, options: any): Promise<any> {
  const controller = new AbortController();
  const timeout = setTimeout(() => controller.abort(), this.timeoutMs);
  // ... fetch with signal
}

---

Using Your Custom Endpoint

Once registered, use your custom models exactly like any OpenAI-compatible model:

curl -X POST https://your-llmapi.example.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3:8b",
    "messages": [{"role": "user", "content": "Hello from FreeLLMAPI"}],
    "temperature": 0.7
  }'

The server executes this resolution chain:

1. Looks up "llama3:8b" in the models table, finding platform = "custom" and key_id = 23
2. Retrieves the api_keys row for ID 23, getting base_url = "http://localhost:11434/v1" and key = "no-key"
3. Calls resolveProvider(Platform.CUSTOM, "http://localhost:11434/v1")
4. The OpenAICompatProvider forwards the request to http://localhost:11434/v1/chat/completions

---

Managing Custom Endpoints

List All Custom Endpoints

curl https://your-llmapi.example.com/keys/custom

Returns all custom configurations with masked keys and their associated models.

Delete a Custom Endpoint

curl -X DELETE https://your-llmapi.example.com/keys/custom/23

This cascades to delete associated models from the models table due to the foreign key constraint defined in server/src/db/migrations.ts.

---

Database Schema for Custom Providers

The migration in server/src/db/migrations.ts defines the schema supporting custom endpoints:

table.enum('platform', ['groq', 'mistral', 'openrouter', 'custom']).notNullable();
table.text('base_url').nullable(); // For custom OpenAI-compatible endpoints

The base_url column is nullable for built-in providers but required for custom providers. The Platform enum in shared/types.ts includes CUSTOM = 'custom', ensuring type safety across the codebase.

---

Summary

• Single POST to /keys/custom registers any OpenAI-compatible endpoint with FreeLLMAPI
• Dynamic provider instantiation means no server restarts or configuration file edits required
• Automatic routing via resolveProvider() in server/src/providers/index.ts handles request forwarding
• Flexible authentication supports both API-key and keyless services through the OpenAICompatProvider.authHeader() method
• Extended timeouts (30 seconds) accommodate slower self-hosted inference compared to cloud providers

---

Frequently Asked Questions

Can I register multiple custom endpoints?

Yes. Each unique baseUrl creates a distinct entry in the api_keys table with its own key_id. You can register the same model name on different endpoints—FreeLLMAPI routes based on the specific model-to-key mapping stored in the database.

What if my custom endpoint uses a different path structure?

The OpenAICompatProvider expects standard OpenAI API paths (/v1/chat/completions, /v1/models). If your service uses different paths, you may need to proxy it or modify the baseUrl to include any prefix paths before the standard /v1/ segment.

How does FreeLLMAPI handle authentication for local services?

When you register with "apiKey": "no-key" or omit the field entirely, the provider's authHeader() method returns an empty object, and requests are sent without an Authorization header. This matches the behavior of local inference servers like Ollama or text-generation-inference in unauthenticated mode.

Can I update an existing custom endpoint's models?

Yes. Sending another POST /keys/custom request with the same baseUrl updates the existing api_keys row and adds or updates models in the models table. The handler in server/src/routes/keys.ts checks for existing entries and performs updates rather than duplicate inserts.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}