How to Integrate Additional LLM Providers into BettaFish: A Complete Guide

Integrate additional LLM providers into BettaFish by extending the Pydantic settings in config.py with your provider's API key, base URL, and model name, then either configure them as the primary model via environment variables or append them to the rescue fallback chain in the engine agent.

BettaFish centralizes all Large Language Model interactions behind a minimal wrapper class, making it straightforward to extend beyond the default configuration. The architecture uses a consistent three-parameter pattern across all four engines—Report, Query, Media, and Insight—allowing you to plug in any OpenAI-compatible endpoint or custom SDK implementation according to the bettafish source code.

Understanding the LLMClient Architecture

All LLM interactions in BettaFish are isolated behind the LLMClient class defined in ReportEngine/llms/base.py. This wrapper provides a unified interface for both synchronous and streaming completions across every engine.

Core Methods

The LLMClient exposes three public methods that engines consume:

• invoke(system_prompt, user_prompt, **kwargs) – Executes a one-shot, non-streaming completion and returns the full response string.
• stream_invoke(system_prompt, user_prompt, **kwargs) – Returns a generator that yields incremental delta chunks for real-time streaming.
• stream_invoke_to_string(...) – A convenience wrapper that safely concatenates streaming chunks into a final string.

Under the hood, these methods call openai.OpenAI(...).chat.completions.create(), meaning any provider must accept the standard OpenAI Chat Completion payload format.

Engine Initialization Pattern

Each engine instantiates its primary client in the _initialize_llm() method. In ReportEngine/agent.py (lines 50-62), the initialization follows this pattern:

def _initialize_llm(self) -> LLMClient:
    return LLMClient(
        api_key=self.config.REPORT_ENGINE_API_KEY,
        model_name=self.config.REPORT_ENGINE_MODEL_NAME,
        base_url=self.config.REPORT_ENGINE_BASE_URL,
    )

All other engines follow identical conventions, reading their own prefixed environment variables (e.g., QUERY_ENGINE_*, MEDIA_ENGINE_*, INSIGHT_ENGINE_*). For resilience, engines also build a rescue chain in _initialize_rescue_llms() (lines 63-91) that attempts fallback providers if the primary model fails.

Step-by-Step Integration Guide

Adding a new LLM provider requires three components: configuration schema updates, engine wiring, and compatibility verification.

1. Extend the Configuration Schema

BettaFish uses pydantic-settings in config.py to manage environment variables. Each provider requires three fields:

• *_API_KEY – Authentication token for the provider
• *_BASE_URL – Optional custom endpoint for OpenAI-compatible APIs
• *_MODEL_NAME – Model identifier string

To add a provider like Anthropic Claude, append these fields to config.py:

from pydantic import Field
from typing import Optional

class Settings(BaseSettings):
    # Existing configurations...

    
    CLAUDE_API_KEY: Optional[str] = Field(
        None, description="Anthropic API key"
    )
    CLAUDE_BASE_URL: Optional[str] = Field(
        "https://api.anthropic.com/v1",
        description="OpenAI-compatible endpoint for Claude"
    )
    CLAUDE_MODEL_NAME: str = Field(
        "claude-3-5-sonnet-20240620",
        description="Claude model identifier"
    )

The fields automatically load from environment variables matching the attribute names (e.g., CLAUDE_API_KEY). No additional loader code is required.

2. Wire the Provider into an Engine

You have two integration options: setting the provider as the primary model or adding it to the rescue fallback chain.

Option A: Configure as Primary Model

Update the environment variables for your target engine. For the ReportEngine, set:

export REPORT_ENGINE_API_KEY=your_claude_key
export REPORT_ENGINE_MODEL_NAME=claude-3-5-sonnet-20240620
export REPORT_ENGINE_BASE_URL=https://api.anthropic.com/v1

Because ReportEngine/agent.py reads the REPORT_ENGINE_* values dynamically, the existing _initialize_llm() method automatically instantiates an LLMClient pointing to Claude.

Option B: Add to Rescue Fallback Chain

To use the new provider as a backup, modify _initialize_rescue_llms() in ReportEngine/agent.py (lines 63-91). Append your provider to the fallback_specs list before the loop creates clients:

fallback_specs.append(
    ("claude_engine",
     settings.CLAUDE_API_KEY,
     settings.CLAUDE_MODEL_NAME,
     settings.CLAUDE_BASE_URL,
    )
)

The rescue chain now attempts providers in this order: Report → Forum → Insight → Media → Claude. This list is also consumed by create_llm_repair_functions() in ReportEngine/utils/chart_repair_api.py, so your new provider automatically becomes available for JSON/chart validation repairs.

3. Ensure OpenAI Compatibility

LLMClient strictly uses the OpenAI Chat Completion schema (messages list with roles, optional stream=True). Most modern providers—including Anthropic via compatibility proxies, Azure OpenAI, and Ollama—support this format.

If your provider uses a different schema, subclass LLMClient and override invoke() and stream_invoke():

from typing import Generator
import requests

class CustomLLMClient:
    def __init__(self, api_key: str, model_name: str, base_url: str):
        self.api_key = api_key
        self.model_name = model_name
        self.base_url = base_url

    def invoke(self, system_prompt: str, user_prompt: str, **kwargs) -> str:
        payload = {
            "model": self.model_name,
            "messages": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": user_prompt},
            ],
        }
        resp = requests.post(
            f"{self.base_url}/v1/chat/completions",
            json=payload,
            headers={"Authorization": f"Bearer {self.api_key}"},
            timeout=30,
        )
        resp.raise_for_status()
        return resp.json()["choices"][0]["message"]["content"]

    def stream_invoke(self, system_prompt: str, user_prompt: str, **kwargs) -> Generator[str, None, None]:
        # Implement SSE streaming logic here

        raise NotImplementedError("Streaming not implemented for this provider")

Inject this custom class into the rescue list in ReportEngine/agent.py:

from custom_llm import CustomLLMClient

client = CustomLLMClient(
    api_key=settings.CUSTOM_API_KEY,
    model_name=settings.CUSTOM_MODEL_NAME,
    base_url=settings.CUSTOM_BASE_URL,
)
clients.append(("custom_engine", client))

Key Files for LLM Integration

When extending BettaFish to support additional providers, these files contain the critical logic:

• config.py – Defines the Pydantic settings model. Add new *_API_KEY, *_BASE_URL, and *_MODEL_NAME fields here.
• ReportEngine/llms/base.py – Contains the LLMClient class implementation using the OpenAI SDK.
• ReportEngine/agent.py – Houses _initialize_llm() (primary client) and _initialize_rescue_llms() (fallback chain) at lines 50-91.
• ReportEngine/utils/chart_repair_api.py – Generates repair functions that automatically inherit providers from the rescue chain via create_llm_repair_functions().

Summary

• BettaFish uses LLMClient in ReportEngine/llms/base.py to abstract all LLM calls behind three methods: invoke(), stream_invoke(), and stream_invoke_to_string().
• Configuration requires three variables (API_KEY, BASE_URL, MODEL_NAME) defined in config.py and loaded from environment variables.
• Primary models are set by exporting the engine-specific variables (e.g., REPORT_ENGINE_API_KEY) before runtime.
• Fallback providers are added to the fallback_specs list in _initialize_rescue_llms() and automatically propagate to chart repair logic.
• OpenAI compatibility is required unless you implement a custom client class with matching method signatures.

Frequently Asked Questions

Can I integrate Anthropic Claude without modifying Python code?

Yes. If you use an OpenAI-compatible proxy endpoint for Claude, simply set the REPORT_ENGINE_API_KEY, REPORT_ENGINE_MODEL_NAME, and REPORT_ENGINE_BASE_URL environment variables. The existing _initialize_llm() method in ReportEngine/agent.py will instantiate the client using your Claude credentials without requiring code changes.

How does the rescue fallback chain prioritize providers?

The rescue chain executes in the order providers are appended to fallback_specs in _initialize_rescue_llms(). By default, BettaFish attempts Report, Forum, Insight, and Media engines sequentially. When you append a new provider to this list, it becomes the final fallback option. Each client attempts the LLM call, and if it fails, the chain proceeds to the next provider.

What if my LLM provider does not support the OpenAI API format?

You must implement a custom client class with invoke() and stream_invoke() methods that translate between OpenAI-style payloads and your provider's native format. Instantiate this class and inject it into the rescue chain in ReportEngine/agent.py. Ensure the method signatures match LLMClient to maintain compatibility with the engine's error handling.

Will new providers automatically work with BettaFish's chart repair functionality?

Yes. The create_llm_repair_functions() utility in ReportEngine/utils/chart_repair_api.py iterates through the same fallback_specs list used by the rescue chain. When you add a provider to _initialize_rescue_llms(), it automatically becomes available for LLM-based JSON repair during chart validation without additional configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}