Best Practices for Using FreeLLM API: Security, Routing & Performance Guide

Use model: "auto", enable sticky sessions, and store provider keys encrypted with AES‑256‑GCM to maximize reliability and security.

FreeLLM API is an open‑source, OpenAI‑compatible router that aggregates 18 free‑tier LLM providers and 161 models behind a single /v1 endpoint. According to the tashfeenahmed/freellmapi source code, following architectural best practices ensures your deployment remains secure, performant, and resilient to provider outages.

---

Secure API Key Management

Encrypt Provider Keys at Rest

The router encrypts every upstream provider key using AES‑256‑GCM before writing to SQLite. This guarantees that a compromised host cannot read raw API keys. In server/src/lib/crypto.ts, the encryption envelope includes a nonce and authentication tag, ensuring confidentiality and integrity.

Isolate Apps with a Unified API Key

Clients authenticate with a freellmapi‑… bearer token, not the upstream provider keys. This unified API key isolates downstream services from your secrets. Configure it in your environment file:

# .env - generated on first run, never commit to version control

FREELLMAPI_KEY=freellmapi-your-unified-key

Reference the .env.example file for the full variable list.

---

Optimize Model Selection and Routing

Prefer model: "auto" for Intelligent Routing

Setting model: "auto" delegates model selection to the router. The logic in server/src/services/router.ts evaluates health status, rate limits, and your configured fallback chain to pick the best available model automatically.

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3001/v1",
    api_key="freellmapi-your-unified-key",
)

resp = client.chat.completions.create(
    model="auto",  # Router selects best available model

    messages=[{"role": "user", "content": "Summarise quantum computing."}],
)

print(resp.choices[0].message.content)
print("Routed via:", resp.headers.get("x-routed-via"))

Configure Your Fallback Chain

The router respects a user‑defined fallback order. Place your most capable (and least rate‑limited) models at the top using the dashboard Keys → Fallback Chain page. This directly influences the selection logic in router.ts when automatic fall‑over occurs on 429/5xx responses.

---

Handle Session Continuity and Context

Enable Sticky Sessions

The router remembers the selected model for 30 minutes per session, preventing mid‑conversation model switches that can cause hallucination spikes. This behavior is documented in the README features section.

Turn On Context Handoff for Model Switches

When a model switch is unavoidable, set FREELLMAPI_CONTEXT_HANDOFF=on_model_switch in your environment. This injects a concise system message informing the new model that it is taking over an ongoing task, preserving conversational continuity.

---

Monitor Health and Rate Limits

Leverage Built‑In Health Probes

The server/src/services/health.ts module runs periodic probes that classify each key as healthy, rate_limited, invalid, or error. The router uses this status to avoid routing to degraded providers.

Respect Rate Limits with the Ledger

In‑memory RPM/RPD/TPM/TPD counters in server/src/services/ratelimit.ts enforce caps and cooldowns per key. The router automatically cools down failing keys and retries others, but you should monitor the analytics dashboard to identify patterns.

Rotate Keys Regularly

Replace invalid or rate‑limited keys via the dashboard Keys page. The router handles hot‑swapping without downtime.

---

Keep the Model Catalog Current

The router pulls a signed catalog from freellmapi.co twice daily, as implemented in server/src/services/router.ts. For production deployments:

• Run the latest version to receive catalog updates
• Enable the Premium (live catalog) feed to access new models the day they ship

This ensures you have access to the full 161‑model pool and any newly added providers.

---

Deployment and Network Security

Run in a Trusted Environment

Bind the service appropriately for your network topology:

# For local-only access

HOST_BIND=127.0.0.1

# For LAN access (protect with firewall rules)

HOST_BIND=0.0.0.0

Protect remote access to the unified key with TLS or SSH tunneling. The Docker setup in the README provides containerized isolation.

---

Advanced Usage Patterns

Streaming Responses

Stream tokens efficiently without managing model selection manually:

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Stream me a haiku about SQLite."}],
    stream=True,
)

for chunk in stream:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Tool Calling Workflows

The router supports OpenAI‑compatible tool calling across providers:

tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get current weather for a city.",
        "parameters": {
            "type": "object",
            "properties": {"city": {"type": "string"}},
            "required": ["city"],
        },
    },
}]

# Step 1: Model requests tool call

first = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "What's the weather in Karachi?"}],
    tools=tools,
    tool_choice="required",
)

call = first.choices[0].message.tool_calls[0]

# Step 2: Execute tool and continue conversation

final = client.chat.completions.create(
    model="auto",
    messages=[
        {"role": "user", "content": "What's the weather in Karachi?"},
        first.choices[0].message,
        {"role": "tool", "tool_call_id": call.id,
         "content": '{"temp_c": 32, "cond": "sunny"}'},
    ],
    tools=tools,
)

print(final.choices[0].message.content)

Vision and Embeddings

Vision requests follow the same model: "auto" pattern:

resp = client.chat.completions.create(
    model="auto",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url",
             "image_url": {"url": "data:image/png;base64,<BASE64_DATA>"}},
        ],
    }],
)

For embeddings, use the auto family selector:

emb = client.embeddings.create(
    model="auto",  # Defaults to gemini-embedding-001 family

    input=["text to embed", "more text"],
)

print(f"{len(emb.data)} vectors of {len(emb.data[0].embedding)} dimensions")

---

Extending with Provider Adapters

When adding new providers, follow the template in server/src/providers/openai-compat.ts. Each adapter must implement chatCompletion() and streamChatCompletion() methods to integrate with the router's provider-agnostic interface.

---

Summary

• Encrypt keys using AES‑256‑GCM via server/src/lib/crypto.ts and isolate apps with a unified bearer token
• Use model: "auto" to let server/src/services/router.ts select optimal models based on health and rate limits
• Configure fallback chains in the dashboard, placing reliable models first
• Enable sticky sessions and context handoff (FREELLMAPI_CONTEXT_HANDOFF=on_model_switch) for conversation continuity
• Monitor health via server/src/services/health.ts and rate limits via server/src/services/ratelimit.ts
• Rotate keys regularly and keep catalogs fresh by updating to the latest version
• Secure deployment with proper HOST_BIND settings and TLS for remote access

---

Frequently Asked Questions

How does FreeLLM API secure my provider API keys?

FreeLLM API encrypts all provider keys using AES‑256‑GCM before storing them in SQLite, as implemented in server/src/lib/crypto.ts. This envelope encryption ensures that even if the host is compromised, the raw keys remain inaccessible without the decryption key.

What happens when a provider returns a 429 or 5xx error?

The router in server/src/services/router.ts automatically falls back to the next healthy model in your configured chain. The health service tracks key status in real-time, and the rate-limit ledger manages cooldowns to prevent hammering failing endpoints.

Can I force a specific model instead of using "auto"?

Yes, you can pass any specific model name from the 161-model catalog. However, using model: "auto" is recommended because it leverages health checks, rate-limit awareness, and your fallback preferences to maximize availability and quality.

How do I add a new provider to the router?

Create a new adapter in server/src/providers/ following the pattern in openai-compat.ts. Each adapter must implement chatCompletion() and streamChatCompletion() methods that conform to the OpenAI-compatible interface expected by the router.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}