How to Configure Ollama or Local LLM Backends for Offline DB-GPT Deployments

DB-GPT supports fully offline deployments by routing LLM calls to local model servers like Ollama or llama.cpp through provider-specific TOML configuration files that map to client adapters in the packages/dbgpt-core/src/dbgpt/model/ directory.

DB-GPT enables complete air-gapped operation by connecting to locally hosted large language models instead of cloud APIs. This configuration leverages the provider-based plugin system implemented in the eosphoros-ai/DB-GPT repository, allowing you to switch between Ollama, llama.cpp, or vLLM without code changes. By editing the TOML configuration files and pointing to local endpoints, you can run DB-GPT entirely offline while maintaining full chat, RAG, and database interaction capabilities.

Install and Start Your Local Model Server

Before configuring DB-GPT, you must have a local model server running. DB-GPT supports two primary patterns for local LLM access: HTTP-based proxy servers (Ollama) and direct model file loading (llama.cpp).

Ollama Setup (HTTP Proxy Mode)

Ollama provides a lightweight HTTP API that serves quantized models without requiring complex GPU driver configurations. Install and start the server:

# Install Ollama (Linux/macOS)

curl -fsSL https://ollama.com/install.sh | sh

# Pull a model (e.g., deepseek-r1)

ollama pull deepseek-r1:1.5b

# Start the Ollama API server (default port 11434)

ollama serve &

The server listens on http://localhost:11434 by default. DB-GPT's OllamaLLMClient (defined in packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py) connects to this endpoint to route all completion requests.

Llama.cpp Setup (Direct Model Loading)

For maximal performance with CPU-only or GPU-accelerated inference, use llama.cpp to load GGUF model files directly:

# Install the Python server helper

pip install llama-cpp-server-py

# Download a GGUF model and place it in your models directory

# Example: models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf

# Launch the server

python -m llama_cpp_server_py \
    --model models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf \
    --host 0.0.0.0 --port 8000 &

This exposes a compatible /chat endpoint that the LlamaCppLLMClient (located in packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py) consumes.

Configure DB-GPT for Local Backends

DB-GPT loads model definitions from TOML configuration files in the configs/ directory. The system uses the provider field to determine which client class to instantiate via the registry in dbgpt/model/proxy/base.py.

Ollama Proxy Configuration

Create or edit configs/dbgpt-proxy-ollama.toml to point to your local Ollama instance:

[system]
language = "${env:DBGPT_LANG:-en}"
encrypt_key = "your_secret_key"

[service.web]
host = "0.0.0.0"
port = 5670

[rag.storage.vector]
type = "chroma"
persist_path = "pilot/data"

[models]
[[models.llms]]
name = "deepseek-r1:1.5b"
provider = "proxy/ollama"
api_base = "http://localhost:11434"
api_key = ""

The provider = "proxy/ollama" string triggers DB-GPT to instantiate OllamaLLMClient. The api_base parameter points to the local server; see lines 41-48 in ollama.py for the default dataclass parameters.

Llama.cpp Local Configuration

For direct file-based loading without a separate server process, use the llama.cpp provider:

[[models.llms]]
name = "DeepSeek-R1-Distill-Qwen-1.5B"
provider = "llama.cpp"
path = "models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf"

When provider = "llama.cpp", DB-GPT loads LlamaCppLLMClient and passes the path field directly to the underlying C++ library. The model path can be absolute or relative to the DB-GPT working directory.

Launch and Verify the Offline Deployment

Starting DB-GPT with Custom Config

Activate your virtual environment and launch the application with your chosen configuration:

source .venv/bin/activate

# For Ollama backend

python -m dbgpt_app.main --config configs/dbgpt-proxy-ollama.toml

# For llama.cpp backend

python -m dbgpt_app.main --config configs/dbgpt-local-llama-cpp.toml

The startup sequence performs the following operations:

1. Parses the TOML configuration via dbgpt/configs/model_config.py
2. Resolves the provider string to a client class using register_proxy_model_adapter
3. Instantiates the client through OllamaLLMClient.new_client (lines 82-94 in ollama.py)
4. Forwards all chat and completion calls to the local server

Testing the Local Connection

Verify your offline setup by programmatically calling the configured model:

from dbgpt.core import ModelRequest
from dbgpt.model.proxy.base import ProxyModel

# Load the proxy model defined in the TOML config

proxy_model = ProxyModel.from_config_name("deepseek-r1:1.5b")
req = ModelRequest(prompt="Explain quantum entanglement in one sentence.")

# Stream the response from the local backend

for chunk in proxy_model.proxy_llm_client.sync_generate_stream(req):
    print(chunk.text, end="")

If text generates instantly without network latency, your offline backend is correctly wired.

Switching Between Local Backends

DB-GPT's provider-based architecture allows backend switching without code modifications. To migrate from Ollama to llama.cpp (or vLLM, HuggingFace, etc.), simply:

1. Stop the DB-GPT service
2. Change the provider field in your TOML file (e.g., from "proxy/ollama" to "llama.cpp")
3. Update backend-specific parameters (api_base for proxies, path for local files)
4. Restart with python -m dbgpt_app.main --config your-new-config.toml

The registry automatically loads the appropriate adapter class based on the provider string.

Summary

• Ollama proxy mode uses HTTP connections to localhost:11434 and requires the provider = "proxy/ollama" setting in your TOML file.
• Direct local loading via llama.cpp uses provider = "llama.cpp" with a file system path to your GGUF model.
• Key source files include packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py for the Ollama client and packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py for the llama.cpp adapter.
• Configuration files are located in the configs/ directory, with examples at configs/dbgpt-proxy-ollama.toml and configs/dbgpt-local-llama-cpp.toml.
• Launch DB-GPT offline using python -m dbgpt_app.main --config <your-config.toml> after starting your local model server.

Frequently Asked Questions

Can I run DB-GPT completely without internet access?

Yes. Once you have downloaded your model files (for llama.cpp) or pulled models via Ollama while online, DB-GPT operates entirely offline. The api_base for Ollama points to localhost, and the vector storage uses local ChromaDB (type = "chroma") by default, ensuring no external API calls are required.

What is the difference between "proxy/ollama" and "llama.cpp" providers?

The "proxy/ollama" provider uses OllamaLLMClient to send HTTP requests to a running Ollama server (typically on port 11434), similar to how cloud APIs work but locally. The "llama.cpp" provider uses LlamaCppLLMClient to load GGUF model files directly into memory via the llama-cpp-python bindings, bypassing network layers for lower latency.

How do I configure a different local port for Ollama?

Modify the api_base parameter in your TOML configuration file. For example, if you started Ollama on port 8080, set api_base = "http://localhost:8080" in the [[models.llms]] section. The OllamaLLMClient reads this value during initialization (lines 82-94 in ollama.py) and routes all requests to the specified endpoint.

Does DB-GPT support other local backends like vLLM or HuggingFace?

Yes. The provider system supports multiple local inference engines. For vLLM, use provider = "vllm" with the appropriate api_base pointing to your vLLM server. For HuggingFace transformers local loading, use provider = "hf". Each provider maps to a specific client class in packages/dbgpt-core/src/dbgpt/model/, following the same TOML configuration pattern as Ollama and llama.cpp.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}