How to Set Up LiteLLM Provider with Custom Model Endpoints in OpenViking

You configure a custom endpoint in OpenViking by passing the api_base URL to LiteLLMProvider or declaring it in the providers section of your VLM configuration, which automatically routes requests to your self-hosted or private LLM server.

OpenViking uses LiteLLM as a unified wrapper for multiple LLM services, enabling seamless integration with both commercial APIs and private endpoints. The LiteLLMProvider class in the volcengine/OpenViking repository handles the heavy lifting by forwarding your custom api_base directly to LiteLLM while managing environment variables through the provider registry. Whether you are connecting to a self-hosted Moonshot instance or a private OpenAI-compatible server, you only need to supply the endpoint URL and optional authentication credentials.

Understanding the LiteLLM Provider Architecture

The integration relies on three core components that work together to resolve and route requests to custom endpoints.

LiteLLMProvider (bot/vikingbot/providers/litellm_provider.py) implements the LLM interface and manages the actual API calls. It stores your api_base during initialization and forwards it unchanged to LiteLLM during the chat method execution.

Provider Registry (bot/vikingbot/providers/registry.py) maintains ProviderSpec definitions for every supported service. The find_gateway function detects gateway and local providers based on your provider_name, API key prefix, or api_base keyword, ensuring the correct environment variables are set.

VLM Configuration (openviking_cli/utils/config/vlm_config.py) provides the Pydantic model that aggregates settings from your config files. The _build_vlm_config_dict method converts the providers mapping into the dictionary passed to LiteLLMProvider, including your custom api_base and api_key.

Configuring Custom Endpoints via VLMConfig

The simplest way to set up a custom model endpoint is through the YAML configuration file used by the OpenViking CLI.

Add your endpoint details under the providers section:

# config.yaml

vlm:
  model: "kimi-k2.5"
  provider: "moonshot"
  providers:
    moonshot:
      api_key: "my-moonshot-key"
      api_base: "https://my.private.moonshot/api/v1"

When the CLI loads this configuration, VLMConfig._build_vlm_config_dict (lines 39-44) constructs the runtime dictionary:

{
    "model": "kimi-k2.5",
    "temperature": 0.0,
    "max_retries": 2,
    "provider": "moonshot",
    "api_key": "my-moonshot-key",
    "api_base": "https://my.private.moonshot/api/v1",
}

VLMFactory.create then instantiates LiteLLMProvider with these values, automatically routing requests to your private endpoint instead of the default Moonshot API.

Programmatic Setup with LiteLLMProvider

For dynamic configurations or custom applications, instantiate LiteLLMProvider directly with your endpoint parameters:

from bot.vikingbot.providers.litellm_provider import LiteLLMProvider

# Configure for a self-hosted OpenAI-compatible server

custom_provider = LiteLLMProvider(
    api_key="my-selfhosted-key",
    api_base="http://localhost:8000/v1",
    provider_name="openai",
    default_model="gpt-4o-mini",
)

# Execute chat completion

response = await custom_provider.chat(
    messages=[{"role": "user", "content": "What is the weather today?"}],
    model="gpt-4o-mini",
)
print(response.content)

How this works:

1. Initialization (__init__, lines 26-33): Stores api_base as self.api_base and calls _setup_env to configure environment variables.

2. Gateway Detection (find_gateway in registry.py, lines 34-45): Uses your provider_name or api_base keyword to select the correct ProviderSpec, ensuring LiteLLM receives the proper routing prefix.

3. Request Forwarding (chat method, lines 46-49): Injects kwargs["api_base"] = self.api_base into the LiteLLM call, overriding the default base URL with your custom endpoint.

Adding Extra Headers for Custom Authentication

Some private endpoints require additional headers beyond standard API keys. Pass these through the extra_headers parameter:

custom_provider = LiteLLMProvider(
    api_key="my-key",
    api_base="https://my.service/v1",
    extra_headers={"APP-Code": "my-app-code", "X-Custom-Auth": "token"},
    default_model="my-model",
)

response = await custom_provider.chat(
    messages=[{"role": "user", "content": "Explain the term 'LLM'"}],
    model="my-model",
)

The provider injects these headers into the LiteLLM request via kwargs["extra_headers"] = self.extra_headers (lines 50-53 in litellm_provider.py).

How the Registry Resolves Custom Endpoints

The provider registry (bot/vikingbot/providers/registry.py) contains ProviderSpec objects that define environment variable mappings and LiteLLM prefixes for each service. When you specify a custom api_base, the find_gateway function checks detection rules including:

• Provider name matching (e.g., "moonshot", "openai")
• API base keyword detection (e.g., URLs containing "moonshot" trigger the Moonshot spec)

This automatic resolution ensures that the correct environment variables (such as MOONSHOT_API_KEY or OPENAI_API_KEY) are exported via _setup_env (lines 57-78) before LiteLLM executes the request.

Summary

• Declare endpoints in config: Add api_base and api_key to the providers section of VLMConfig for CLI-based usage.
• Instantiate programmatically: Pass api_base directly to LiteLLMProvider for dynamic or embedded applications.
• Automatic resolution: The provider registry (registry.py) detects your endpoint type and sets required environment variables via find_gateway and _setup_env.
• Header support: Use extra_headers for proprietary authentication schemes required by private endpoints.
• Direct forwarding: The chat method passes api_base unchanged to LiteLLM, ensuring requests reach your custom URL.

Frequently Asked Questions

How does OpenViking route requests to my custom API endpoint?

OpenViking routes requests by storing your custom URL in LiteLLMProvider.api_base and forwarding it to LiteLLM as the api_base parameter during the chat call (lines 46-48 in litellm_provider.py). LiteLLM then constructs the full request URL using your endpoint instead of the default provider URL.

Can I use custom endpoints with any LLM provider in the registry?

Yes. The find_gateway function in registry.py (lines 34-45) supports custom endpoints for any defined ProviderSpec. You can override the default api_base for OpenAI, Moonshot, Anthropic, or any other registered provider by specifying the api_base URL in your configuration or constructor.

Where do I store API keys for custom endpoints?

Store API keys in the providers section of your VLMConfig YAML file under the specific provider key (e.g., providers.moonshot.api_key), or pass them directly to the LiteLLMProvider constructor as the api_key parameter. The _setup_env method (lines 57-78) automatically maps these to the correct environment variables expected by LiteLLM.

How do I debug connection issues with custom endpoints?

Enable LiteLLM debugging by removing the litellm.suppress_debug_info = True line in LiteLLMProvider.__init__ (lines 52-55), or set litellm.set_verbose=True in your code. Verify that your api_base includes the correct API version path (typically /v1 for OpenAI-compatible endpoints) and that the provider_name matches a valid entry in registry.py to ensure proper environment variable configuration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}