How to Configure Multiple LLM Providers in DB-GPT Using the SMMF Framework

DB-GPT's SMMF (Service-oriented Multi-model Management Framework) lets you run OpenAI, Azure, vLLM, Ollama, and other LLM providers simultaneously by defining each model in the model_config file, allowing the Model Controller to automatically route requests to the correct backend.

The SMMF framework in DB-GPT abstracts model inference and deployment, enabling a single instance to serve many different LLM back-ends without code changes. By leveraging the model_config configuration and the Model Controller, you can register multiple providers at startup and switch between them at runtime using the Python SDK or REST API. This guide walks through the architecture, configuration syntax, and practical code examples based on the DB-GPT source code.

Understanding the SMMF Architecture

The Service-oriented Multi-model Management Framework separates model definition from consumption through three core components: the configuration registry, the Model Controller, and the DefaultLLMClient.

Model Controller and Model Handles

At startup, dbgpt.model.cluster.controller loads every entry from your configuration file and builds a model handle for each LLM. The BaseModelController class (located in packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py) stores these handles in an internal registry, making all configured models discoverable throughout the application lifecycle.

Each ModelHandle encapsulates the connection details, authentication, and provider-specific settings for one LLM. Because the controller holds all configured models simultaneously, DB-GPT can route requests to OpenAI, a local vLLM instance, and an Ollama server without restarting the service.

DefaultLLMClient and Provider Resolution

When a component—such as a RAG service, agent, or AWEL flow—needs to generate text, it calls DefaultLLMClient. This façade (defined in packages/dbgpt-core/src/dbgpt/model/__init__.py) receives a model_name argument, queries the Model Controller for the corresponding handle, and returns the correct proxy class (OpenAILLMClient, VllmLLMClient, OllamaLLMClient, etc.) based on the provider field in the configuration.

The client method DefaultLLMClient(worker_manager).generate(...) automatically resolves the correct provider, so the same high-level code works for every LLM regardless of whether it is a cloud API or a local inference server.

Configuring Multiple LLM Providers

You define each LLM as a separate table in the global model_config file using the [[models.llms]] syntax. DB-GPT supports TOML or YAML formats, with TOML being the most common for production deployments.

Configuration File Structure

Add one [[models.llms]] block per provider in your config.toml or dedicated models.toml file:

[[models.llms]]
name = "gpt-4o"
provider = "openai"
model_name = "gpt-4o"
api_key = "${OPENAI_API_KEY}"

[[models.llms]]
name = "llama3-local"
provider = "vllm"
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
path = "/opt/models/llama3"

[[models.llms]]
name = "ollama-mistral"
provider = "ollama"
model_name = "mistral"
api_base = "http://localhost:11434"

Key Configuration Fields

The model_config.py file (located in packages/dbgpt-core/src/dbgpt/configs/model_config.py) defines the schema and defaults for each entry:

• name: A human-readable identifier used in API calls and the Python SDK (e.g., "gpt-4o").
• provider: Determines which proxy implementation to load. Valid values include openai, vllm, ollama, tgi, and others.
• model_name: The actual model identifier understood by the provider's API.
• path: Required for local inference providers like vllm or llama.cpp, specifying the filesystem location of the downloaded weights.
• api_key and api_base: Authentication credentials and endpoint URLs for remote services.

All entries are collected into a list at runtime, so you can declare as many providers as needed without modifying application code.

Runtime Selection and Usage

Once configured, you can list available models and explicitly select one for specific requests using the Python SDK or REST API.

Listing Registered Models via Python

Retrieve metadata for all configured LLMs using the DefaultLLMClient:

from dbgpt.model import DefaultLLMClient
from dbgpt.core import ModelMetadata

# Retrieve the model controller (automatically started with the server)

client = DefaultLLMClient()
metadata: list[ModelMetadata] = client.get_model_metadata()

for m in metadata:
    print(f"{m.name} → {m.provider} ({m.model_name})")

Explicit Provider Selection in RAG Services

Pass a specific model name when initializing downstream services like RAG:

from dbgpt.model import DefaultLLMClient
from dbgpt.rag.service import RAGService

# Use the Ollama model configured earlier

llm_client = DefaultLLMClient(model_name="ollama-mistral")
rag = RAGService(llm_client=llm_client)

answer = rag.query("How does DB-GPT store vector embeddings?")
print(answer)

Switching Models via REST API

Query available models and target specific ones in chat completions:

# Get the list of registered models

curl http://localhost:8100/api/v1/models/types

# Use a specific model in a chat request

curl -X POST http://localhost:8100/api/v1/chat \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama3-local",
        "messages": [{"role":"user","content":"Explain SMMF in one sentence"}]
      }'

Key Source Files and Implementation Details

Understanding the following source files helps when debugging provider-specific issues or extending the framework:

• packages/dbgpt-core/src/dbgpt/configs/model_config.py: Contains default values and Pydantic validation for the [[models.llms]] configuration tables.
• packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py: Implements BaseModelController, which loads the configuration, creates ModelHandle instances, and manages the registration lifecycle.
• packages/dbgpt-core/src/dbgpt/model/__init__.py: Exports DefaultLLMClient, the façade used by all components to obtain the proper proxy client without hardcoding provider logic.
• packages/dbgpt-core/src/dbgpt/model/proxy/llms/: Directory containing concrete implementations for each provider (e.g., openai.py, vllm.py, ollama.py).
• docs/docs/modules/smmf.md: Architectural documentation describing the Service-oriented Multi-model Management Framework design principles.

Summary

• SMMF Architecture: DB-GPT uses a Model Controller (BaseModelController) to register multiple LLM handles at startup, and DefaultLLMClient to route requests to the correct provider proxy.
• Configuration Syntax: Define each provider in [[models.llms]] tables within your TOML config, specifying name, provider, model_name, and credentials or local paths.
• Runtime Flexibility: Components like RAG services and agents can select specific models by name via the Python SDK, while REST API consumers can swap models per request using the model parameter.
• Zero Code Changes: Adding a new provider requires only editing the configuration file and restarting the service; no application code modifications are necessary.

Frequently Asked Questions

What does SMMF stand for in DB-GPT?

SMMF stands for Service-oriented Multi-model Management Framework. It is the architectural layer in DB-GPT that abstracts model inference, allowing a single instance to manage and serve multiple LLM back-ends simultaneously through a unified controller and client interface.

How does DB-GPT know which provider class to use for a model?

DB-GPT inspects the provider field in each [[models.llms]] configuration table. The DefaultLLMClient maps this string to a concrete proxy class—such as OpenAILLMClient, VllmLLMClient, or OllamaLLMClient—located in the model_proxy package, then instantiates the correct client for that provider's API.

Can I use both cloud APIs and local inference servers at the same time?

Yes. The SMMF framework is designed specifically for heterogeneous environments. You can configure OpenAI, Azure, vLLM, Ollama, and other providers in the same configuration file. The Model Controller registers all of them, and you can select between cloud and local models on a per-request basis.

Where do I set the API keys for external providers like OpenAI?

API keys and other sensitive credentials are set in the api_key field of the respective [[models.llms]] table. You can use environment variable interpolation (e.g., api_key = "${OPENAI_API_KEY}") to keep secrets out of version-controlled configuration files, as supported by the model_config.py validation logic.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}