# How to Set Up FreeLLMAPI Self-Hosted: Complete Docker Deployment Guide

> Deploy FreeLLMAPI self-hosted with Docker Compose. Aggregate LLM providers like OpenAI and Anthropic behind a unified API. Get the complete guide now.

- Repository: [Tashfeen/freellmapi](https://github.com/tashfeenahmed/freellmapi)
- Tags: how-to-guide
- Published: 2026-09-04

---

**FreeLLMAPI is a self-hosted gateway that aggregates multiple large-language-model providers (OpenAI, Anthropic, Azure) behind a unified OpenAI-compatible REST API, deployable via Docker Compose using a single environment file.**

Setting up FreeLLMAPI self-hosted gives you a private, unified interface to route requests across multiple LLM providers with intelligent failover and quota management. The `tashfeenahmed/freellmapi` repository provides a containerized Fastify server that exposes standard OpenAI endpoints while the routing engine distributes traffic based on health checks and scoring rules. All configuration lives in a single `.env` file, making deployment reproducible and version-controlled.

## Prerequisites

Before deploying FreeLLMAPI self-hosted, ensure your environment meets the following requirements:

- **Docker Engine** (or Docker Desktop) and **Docker Compose** installed on your host machine. See the official installation guide at [`docs/install/OVERVIEW.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/install/OVERVIEW.md) for platform-specific instructions.
- **API keys** for at least one supported provider (OpenAI, Anthropic, Azure OpenAI, or others).
- **Git** to clone the repository.

## Installation Steps

Deploying FreeLLMAPI involves cloning the repository, configuring environment variables, and starting the Docker containers.

### 1. Clone the Repository

```bash
git clone https://github.com/tashfeenahmed/freellmapi.git
cd freellmapi

```

### 2. Configure Environment Variables

Copy the example environment file and edit it to include your provider API keys:

```bash
cp .env.example .env

```

Edit `.env` and populate the required fields. At minimum, configure one provider:

```dotenv
PORT=8080
OPENAI_API_KEY=sk-your-openai-key-here
ANTHROPIC_API_KEY=sk-your-anthropic-key-here
AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
AZURE_OPENAI_API_KEY=your-azure-key

```

The `.env` file controls all configurable values including rate limits (`RATE_LIMIT_GLOBAL`, `RATE_LIMIT_USER`), model weights, and retention settings.

### 3. Start the Service

Launch the container stack in detached mode:

```bash
docker compose up -d

```

This command pulls the official image `ghcr.io/tashfeenahmed/freellmapi:latest`, mounts your `.env` file, and exposes port **8080** (configurable via the `PORT` variable). The service defined in [`docker-compose.yml`](https://github.com/tashfeenahmed/freellmapi/blob/main/docker-compose.yml) includes a restart policy to ensure high availability.

### 4. Verify the Deployment

Test that the gateway is responding by requesting the models list:

```bash
curl http://localhost:8080/v1/models

```

You should receive a JSON array of available models that the router currently exposes from your configured providers.

## Testing API Calls

Once running, FreeLLMAPI accepts standard OpenAI-compatible requests. Send a chat completion through your local gateway:

```bash
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Tell me a joke"}]
  }'

```

The router logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) processes this request, applies bandit scoring to select the optimal provider, and enforces rate limits defined in [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) before forwarding the call.

## Using the CLI Client

The repository includes a TypeScript CLI for management and diagnostics. Install it globally from the repo root:

```bash
npm install -g .

```

Run health checks and test requests:

```bash
llmapi doctor
llmapi chat "Hello, world!"

```

The CLI entry point resides in [`cli/src/index.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/cli/src/index.ts) and provides commands for key management and service diagnostics.

## Core Architecture Components

Understanding the three main components helps with troubleshooting and customization:

- **Router / Bandit Scoring**: Implemented in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts), this component decides which provider to call for each request based on scoring rules, health checks, and user quotas. See [`docs/architecture/01-routing-and-bandit-scoring.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture/01-routing-and-bandit-scoring.md) for detailed logic.
  
- **Provider Quota Engine**: Located in [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts), this tracks usage per-provider, enforces cool-downs when limits are reached, and forecasts remaining quota. Reference [`docs/architecture/02-quota-and-cooldown-engine.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture/02-quota-and-cooldown-engine.md) for implementation details.
  
- **Server & API**: A Fastify-based HTTP server exposing REST endpoints compatible with OpenAI's specification, with optional streaming support.

## Production Configuration

### Data Persistence

By default, the server maintains no state on disk. To enable request retention, persistent rate-limit counters, or model catalog synchronization, mount a volume to `/app/data` and enable persistence flags in `.env`:

```dotenv
RETENTION_ENABLED=true

```

Details on degraded mode and failover configuration are available in [`docs/architecture/04-degraded-mode-and-failover.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture/04-degraded-mode-and-failover.md).

### Updating the Service

To upgrade to the latest version:

```bash
git pull
docker compose pull
docker compose up -d --force-recreate

```

This pulls the latest image and recreates containers without downtime.

## Summary

- FreeLLMAPI self-hosted deploys via Docker Compose using a single `.env` file for all configuration.
- The gateway aggregates OpenAI, Anthropic, Azure, and other providers behind a unified OpenAI-compatible API endpoint.
- Core routing logic in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) provides intelligent load balancing and failover across backends.
- Rate limiting is enforced through [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts) while quota tracking occurs in [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts).
- Optional data persistence and the `llmapi` CLI client support production operational requirements.

## Frequently Asked Questions

### What providers does FreeLLMAPI support?

FreeLLMAPI supports any provider compatible with the OpenAI API format, including OpenAI, Anthropic, Azure OpenAI, and custom endpoints. Configuration occurs entirely through environment variables in `.env`, allowing you to mix multiple providers simultaneously. The router automatically load-balances requests across healthy providers based on latency and quota availability.

### How does the routing logic select between providers?

The **Bandit Scoring** algorithm implemented in [`server/src/services/router.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/router.ts) evaluates provider health, remaining quota, and historical performance to select the optimal backend for each request. If a provider fails health checks or exceeds its configured quota (tracked in [`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts)), the router automatically fails over to the next available option. See [`docs/architecture/01-routing-and-bandit-scoring.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/architecture/01-routing-and-bandit-scoring.md) for the scoring mathematics.

### Can I deploy FreeLLMAPI without Docker?

While Docker is the officially supported deployment method documented in [`docs/install/OVERVIEW.md`](https://github.com/tashfeenahmed/freellmapi/blob/main/docs/install/OVERVIEW.md), the service is a standard Node.js/Fastify application. You can run it directly using `npm install` and `npm start` from the `server` directory, though you must manually manage environment variables and process management. Docker remains the recommended approach for production stability.

### How do I monitor quota usage and rate limits?

The gateway tracks consumption in real-time through the **Provider Quota Engine** ([`server/src/services/provider-quota.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/provider-quota.ts)). You can inspect current limits via the CLI using `llmapi doctor`, or query the internal health endpoints. Per-user and global rate limits are enforced by [`server/src/services/ratelimit.ts`](https://github.com/tashfeenahmed/freellmapi/blob/main/server/src/services/ratelimit.ts), with breach events logged to stdout for integration with external monitoring systems.