# How to Integrate Additional LLM Providers into BettaFish: A Complete Guide

> Integrate multiple LLM providers into BettaFish easily. Extend Pydantic settings with API keys and URLs for seamless primary or fallback model configuration. Get started now.

- Repository: [BaiFu/bettafish](https://github.com/666ghj/bettafish)
- Tags: how-to-guide
- Published: 2026-02-23

---

**Integrate additional LLM providers into BettaFish by extending the Pydantic settings in [`config.py`](https://github.com/666ghj/bettafish/blob/main/config.py) with your provider's API key, base URL, and model name, then either configure them as the primary model via environment variables or append them to the rescue fallback chain in the engine agent.**

BettaFish centralizes all Large Language Model interactions behind a minimal wrapper class, making it straightforward to extend beyond the default configuration. The architecture uses a consistent three-parameter pattern across all four engines—**Report**, **Query**, **Media**, and **Insight**—allowing you to plug in any OpenAI-compatible endpoint or custom SDK implementation according to the bettafish source code.

## Understanding the LLMClient Architecture

All LLM interactions in BettaFish are isolated behind the **`LLMClient`** class defined in [`ReportEngine/llms/base.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/llms/base.py). This wrapper provides a unified interface for both synchronous and streaming completions across every engine.

### Core Methods

The `LLMClient` exposes three public methods that engines consume:

- **`invoke(system_prompt, user_prompt, **kwargs)`** – Executes a one-shot, non-streaming completion and returns the full response string.
- **`stream_invoke(system_prompt, user_prompt, **kwargs)`** – Returns a generator that yields incremental delta chunks for real-time streaming.
- **`stream_invoke_to_string(...)`** – A convenience wrapper that safely concatenates streaming chunks into a final string.

Under the hood, these methods call `openai.OpenAI(...).chat.completions.create()`, meaning any provider must accept the standard OpenAI Chat Completion payload format.

### Engine Initialization Pattern

Each engine instantiates its primary client in the `_initialize_llm()` method. In [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py) (lines 50-62), the initialization follows this pattern:

```python
def _initialize_llm(self) -> LLMClient:
    return LLMClient(
        api_key=self.config.REPORT_ENGINE_API_KEY,
        model_name=self.config.REPORT_ENGINE_MODEL_NAME,
        base_url=self.config.REPORT_ENGINE_BASE_URL,
    )

```

All other engines follow identical conventions, reading their own prefixed environment variables (e.g., `QUERY_ENGINE_*`, `MEDIA_ENGINE_*`, `INSIGHT_ENGINE_*`). For resilience, engines also build a rescue chain in `_initialize_rescue_llms()` (lines 63-91) that attempts fallback providers if the primary model fails.

## Step-by-Step Integration Guide

Adding a new LLM provider requires three components: configuration schema updates, engine wiring, and compatibility verification.

### 1. Extend the Configuration Schema

BettaFish uses **pydantic-settings** in [`config.py`](https://github.com/666ghj/bettafish/blob/main/config.py) to manage environment variables. Each provider requires three fields:

- **`*_API_KEY`** – Authentication token for the provider
- **`*_BASE_URL`** – Optional custom endpoint for OpenAI-compatible APIs
- **`*_MODEL_NAME`** – Model identifier string

To add a provider like Anthropic Claude, append these fields to [`config.py`](https://github.com/666ghj/bettafish/blob/main/config.py):

```python
from pydantic import Field
from typing import Optional

class Settings(BaseSettings):
    # Existing configurations...

    
    CLAUDE_API_KEY: Optional[str] = Field(
        None, description="Anthropic API key"
    )
    CLAUDE_BASE_URL: Optional[str] = Field(
        "https://api.anthropic.com/v1",
        description="OpenAI-compatible endpoint for Claude"
    )
    CLAUDE_MODEL_NAME: str = Field(
        "claude-3-5-sonnet-20240620",
        description="Claude model identifier"
    )

```

The fields automatically load from environment variables matching the attribute names (e.g., `CLAUDE_API_KEY`). No additional loader code is required.

### 2. Wire the Provider into an Engine

You have two integration options: setting the provider as the primary model or adding it to the rescue fallback chain.

**Option A: Configure as Primary Model**

Update the environment variables for your target engine. For the ReportEngine, set:

```bash
export REPORT_ENGINE_API_KEY=your_claude_key
export REPORT_ENGINE_MODEL_NAME=claude-3-5-sonnet-20240620
export REPORT_ENGINE_BASE_URL=https://api.anthropic.com/v1

```

Because [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py) reads the `REPORT_ENGINE_*` values dynamically, the existing `_initialize_llm()` method automatically instantiates an `LLMClient` pointing to Claude.

**Option B: Add to Rescue Fallback Chain**

To use the new provider as a backup, modify `_initialize_rescue_llms()` in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py) (lines 63-91). Append your provider to the `fallback_specs` list before the loop creates clients:

```python
fallback_specs.append(
    ("claude_engine",
     settings.CLAUDE_API_KEY,
     settings.CLAUDE_MODEL_NAME,
     settings.CLAUDE_BASE_URL,
    )
)

```

The rescue chain now attempts providers in this order: Report → Forum → Insight → Media → Claude. This list is also consumed by `create_llm_repair_functions()` in [`ReportEngine/utils/chart_repair_api.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/utils/chart_repair_api.py), so your new provider automatically becomes available for JSON/chart validation repairs.

### 3. Ensure OpenAI Compatibility

`LLMClient` strictly uses the OpenAI Chat Completion schema (messages list with roles, optional `stream=True`). Most modern providers—including Anthropic via compatibility proxies, Azure OpenAI, and Ollama—support this format.

If your provider uses a different schema, subclass `LLMClient` and override `invoke()` and `stream_invoke()`:

```python
from typing import Generator
import requests

class CustomLLMClient:
    def __init__(self, api_key: str, model_name: str, base_url: str):
        self.api_key = api_key
        self.model_name = model_name
        self.base_url = base_url

    def invoke(self, system_prompt: str, user_prompt: str, **kwargs) -> str:
        payload = {
            "model": self.model_name,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt},
            ],
        }
        resp = requests.post(
            f"{self.base_url}/v1/chat/completions",
            json=payload,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=30,
        )
        resp.raise_for_status()
        return resp.json()["choices"][0]["message"]["content"]

    def stream_invoke(self, system_prompt: str, user_prompt: str, **kwargs) -> Generator[str, None, None]:
        # Implement SSE streaming logic here

        raise NotImplementedError("Streaming not implemented for this provider")

```

Inject this custom class into the rescue list in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py):

```python
from custom_llm import CustomLLMClient

client = CustomLLMClient(
    api_key=settings.CUSTOM_API_KEY,
    model_name=settings.CUSTOM_MODEL_NAME,
    base_url=settings.CUSTOM_BASE_URL,
)
clients.append(("custom_engine", client))

```

## Key Files for LLM Integration

When extending BettaFish to support additional providers, these files contain the critical logic:

- **[`config.py`](https://github.com/666ghj/bettafish/blob/main/config.py)** – Defines the Pydantic settings model. Add new `*_API_KEY`, `*_BASE_URL`, and `*_MODEL_NAME` fields here.
- **[`ReportEngine/llms/base.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/llms/base.py)** – Contains the `LLMClient` class implementation using the OpenAI SDK.
- **[`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py)** – Houses `_initialize_llm()` (primary client) and `_initialize_rescue_llms()` (fallback chain) at lines 50-91.
- **[`ReportEngine/utils/chart_repair_api.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/utils/chart_repair_api.py)** – Generates repair functions that automatically inherit providers from the rescue chain via `create_llm_repair_functions()`.

## Summary

- **BettaFish uses `LLMClient`** in [`ReportEngine/llms/base.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/llms/base.py) to abstract all LLM calls behind three methods: `invoke()`, `stream_invoke()`, and `stream_invoke_to_string()`.
- **Configuration requires three variables** (`API_KEY`, `BASE_URL`, `MODEL_NAME`) defined in [`config.py`](https://github.com/666ghj/bettafish/blob/main/config.py) and loaded from environment variables.
- **Primary models** are set by exporting the engine-specific variables (e.g., `REPORT_ENGINE_API_KEY`) before runtime.
- **Fallback providers** are added to the `fallback_specs` list in `_initialize_rescue_llms()` and automatically propagate to chart repair logic.
- **OpenAI compatibility** is required unless you implement a custom client class with matching method signatures.

## Frequently Asked Questions

### Can I integrate Anthropic Claude without modifying Python code?

**Yes.** If you use an OpenAI-compatible proxy endpoint for Claude, simply set the `REPORT_ENGINE_API_KEY`, `REPORT_ENGINE_MODEL_NAME`, and `REPORT_ENGINE_BASE_URL` environment variables. The existing `_initialize_llm()` method in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py) will instantiate the client using your Claude credentials without requiring code changes.

### How does the rescue fallback chain prioritize providers?

The rescue chain executes in the order providers are appended to `fallback_specs` in `_initialize_rescue_llms()`. By default, BettaFish attempts Report, Forum, Insight, and Media engines sequentially. When you append a new provider to this list, it becomes the final fallback option. Each client attempts the LLM call, and if it fails, the chain proceeds to the next provider.

### What if my LLM provider does not support the OpenAI API format?

You must implement a custom client class with `invoke()` and `stream_invoke()` methods that translate between OpenAI-style payloads and your provider's native format. Instantiate this class and inject it into the rescue chain in [`ReportEngine/agent.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/agent.py). Ensure the method signatures match `LLMClient` to maintain compatibility with the engine's error handling.

### Will new providers automatically work with BettaFish's chart repair functionality?

**Yes.** The `create_llm_repair_functions()` utility in [`ReportEngine/utils/chart_repair_api.py`](https://github.com/666ghj/bettafish/blob/main/ReportEngine/utils/chart_repair_api.py) iterates through the same `fallback_specs` list used by the rescue chain. When you add a provider to `_initialize_rescue_llms()`, it automatically becomes available for LLM-based JSON repair during chart validation without additional configuration.