# How to Configure Multiple LLM Providers in DB-GPT Using the SMMF Framework

> Learn how to configure multiple LLM providers like OpenAI Azure vLLM and Ollama in DB-GPT using the efficient SMMF framework for seamless model management and request routing.

- Repository: [eosphoros/DB-GPT](https://github.com/eosphoros-ai/db-gpt)
- Tags: how-to-guide
- Published: 2026-02-23

---

**DB-GPT's SMMF (Service-oriented Multi-model Management Framework) lets you run OpenAI, Azure, vLLM, Ollama, and other LLM providers simultaneously by defining each model in the `model_config` file, allowing the Model Controller to automatically route requests to the correct backend.**

The SMMF framework in DB-GPT abstracts model inference and deployment, enabling a single instance to serve many different LLM back-ends without code changes. By leveraging the `model_config` configuration and the Model Controller, you can register multiple providers at startup and switch between them at runtime using the Python SDK or REST API. This guide walks through the architecture, configuration syntax, and practical code examples based on the DB-GPT source code.

## Understanding the SMMF Architecture

The Service-oriented Multi-model Management Framework separates model definition from consumption through three core components: the configuration registry, the Model Controller, and the DefaultLLMClient.

### Model Controller and Model Handles

At startup, `dbgpt.model.cluster.controller` loads every entry from your configuration file and builds a **model handle** for each LLM. The `BaseModelController` class (located in [`packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py)) stores these handles in an internal registry, making all configured models discoverable throughout the application lifecycle.

Each `ModelHandle` encapsulates the connection details, authentication, and provider-specific settings for one LLM. Because the controller holds **all** configured models simultaneously, DB-GPT can route requests to OpenAI, a local vLLM instance, and an Ollama server without restarting the service.

### DefaultLLMClient and Provider Resolution

When a component—such as a RAG service, agent, or AWEL flow—needs to generate text, it calls `DefaultLLMClient`. This façade (defined in [`packages/dbgpt-core/src/dbgpt/model/__init__.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/__init__.py)) receives a `model_name` argument, queries the Model Controller for the corresponding handle, and returns the correct proxy class (`OpenAILLMClient`, `VllmLLMClient`, `OllamaLLMClient`, etc.) based on the `provider` field in the configuration.

The client method `DefaultLLMClient(worker_manager).generate(...)` automatically resolves the correct provider, so the same high-level code works for every LLM regardless of whether it is a cloud API or a local inference server.

## Configuring Multiple LLM Providers

You define each LLM as a separate table in the global `model_config` file using the `[[models.llms]]` syntax. DB-GPT supports TOML or YAML formats, with TOML being the most common for production deployments.

### Configuration File Structure

Add one `[[models.llms]]` block per provider in your [`config.toml`](https://github.com/eosphoros-ai/DB-GPT/blob/main/config.toml) or dedicated [`models.toml`](https://github.com/eosphoros-ai/DB-GPT/blob/main/models.toml) file:

```toml
[[models.llms]]
name = "gpt-4o"
provider = "openai"
model_name = "gpt-4o"
api_key = "${OPENAI_API_KEY}"

[[models.llms]]
name = "llama3-local"
provider = "vllm"
model_name = "meta-llama/Meta-Llama-3-8B-Instruct"
path = "/opt/models/llama3"

[[models.llms]]
name = "ollama-mistral"
provider = "ollama"
model_name = "mistral"
api_base = "http://localhost:11434"

```

### Key Configuration Fields

The [`model_config.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/model_config.py) file (located in [`packages/dbgpt-core/src/dbgpt/configs/model_config.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/configs/model_config.py)) defines the schema and defaults for each entry:

- **name**: A human-readable identifier used in API calls and the Python SDK (e.g., `"gpt-4o"`).
- **provider**: Determines which proxy implementation to load. Valid values include `openai`, `vllm`, `ollama`, `tgi`, and others.
- **model_name**: The actual model identifier understood by the provider's API.
- **path**: Required for local inference providers like `vllm` or [`llama.cpp`](https://github.com/eosphoros-ai/DB-GPT/blob/main/llama.cpp), specifying the filesystem location of the downloaded weights.
- **api_key** and **api_base**: Authentication credentials and endpoint URLs for remote services.

All entries are collected into a list at runtime, so you can declare as many providers as needed without modifying application code.

## Runtime Selection and Usage

Once configured, you can list available models and explicitly select one for specific requests using the Python SDK or REST API.

### Listing Registered Models via Python

Retrieve metadata for all configured LLMs using the `DefaultLLMClient`:

```python
from dbgpt.model import DefaultLLMClient
from dbgpt.core import ModelMetadata

# Retrieve the model controller (automatically started with the server)

client = DefaultLLMClient()
metadata: list[ModelMetadata] = client.get_model_metadata()

for m in metadata:
    print(f"{m.name} → {m.provider} ({m.model_name})")

```

### Explicit Provider Selection in RAG Services

Pass a specific model name when initializing downstream services like RAG:

```python
from dbgpt.model import DefaultLLMClient
from dbgpt.rag.service import RAGService

# Use the Ollama model configured earlier

llm_client = DefaultLLMClient(model_name="ollama-mistral")
rag = RAGService(llm_client=llm_client)

answer = rag.query("How does DB-GPT store vector embeddings?")
print(answer)

```

### Switching Models via REST API

Query available models and target specific ones in chat completions:

```bash

# Get the list of registered models

curl http://localhost:8100/api/v1/models/types

# Use a specific model in a chat request

curl -X POST http://localhost:8100/api/v1/chat \
  -H "Content-Type: application/json" \
  -d '{
        "model": "llama3-local",
        "messages": [{"role":"user","content":"Explain SMMF in one sentence"}]
      }'

```

## Key Source Files and Implementation Details

Understanding the following source files helps when debugging provider-specific issues or extending the framework:

- **[`packages/dbgpt-core/src/dbgpt/configs/model_config.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/configs/model_config.py)**: Contains default values and Pydantic validation for the `[[models.llms]]` configuration tables.
- **[`packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/cluster/controller/controller.py)**: Implements `BaseModelController`, which loads the configuration, creates `ModelHandle` instances, and manages the registration lifecycle.
- **[`packages/dbgpt-core/src/dbgpt/model/__init__.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/__init__.py)**: Exports `DefaultLLMClient`, the façade used by all components to obtain the proper proxy client without hardcoding provider logic.
- **`packages/dbgpt-core/src/dbgpt/model/proxy/llms/`**: Directory containing concrete implementations for each provider (e.g., [`openai.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/openai.py), [`vllm.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/vllm.py), [`ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/ollama.py)).
- **[`docs/docs/modules/smmf.md`](https://github.com/eosphoros-ai/DB-GPT/blob/main/docs/docs/modules/smmf.md)**: Architectural documentation describing the Service-oriented Multi-model Management Framework design principles.

## Summary

- **SMMF Architecture**: DB-GPT uses a Model Controller (`BaseModelController`) to register multiple LLM handles at startup, and `DefaultLLMClient` to route requests to the correct provider proxy.
- **Configuration Syntax**: Define each provider in `[[models.llms]]` tables within your TOML config, specifying `name`, `provider`, `model_name`, and credentials or local paths.
- **Runtime Flexibility**: Components like RAG services and agents can select specific models by name via the Python SDK, while REST API consumers can swap models per request using the `model` parameter.
- **Zero Code Changes**: Adding a new provider requires only editing the configuration file and restarting the service; no application code modifications are necessary.

## Frequently Asked Questions

### What does SMMF stand for in DB-GPT?

SMMF stands for **Service-oriented Multi-model Management Framework**. It is the architectural layer in DB-GPT that abstracts model inference, allowing a single instance to manage and serve multiple LLM back-ends simultaneously through a unified controller and client interface.

### How does DB-GPT know which provider class to use for a model?

DB-GPT inspects the `provider` field in each `[[models.llms]]` configuration table. The `DefaultLLMClient` maps this string to a concrete proxy class—such as `OpenAILLMClient`, `VllmLLMClient`, or `OllamaLLMClient`—located in the `model_proxy` package, then instantiates the correct client for that provider's API.

### Can I use both cloud APIs and local inference servers at the same time?

Yes. The SMMF framework is designed specifically for heterogeneous environments. You can configure OpenAI, Azure, vLLM, Ollama, and other providers in the same configuration file. The Model Controller registers all of them, and you can select between cloud and local models on a per-request basis.

### Where do I set the API keys for external providers like OpenAI?

API keys and other sensitive credentials are set in the `api_key` field of the respective `[[models.llms]]` table. You can use environment variable interpolation (e.g., `api_key = "${OPENAI_API_KEY}"`) to keep secrets out of version-controlled configuration files, as supported by the [`model_config.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/model_config.py) validation logic.