# How to Configure Ollama or Local LLM Backends for Offline DB-GPT Deployments

> Configure Ollama or local LLM backends for offline DB-GPT deployments. Learn how to route LLM calls to local model servers using TOML config files for seamless integration.

- Repository: [eosphoros/DB-GPT](https://github.com/eosphoros-ai/db-gpt)
- Tags: how-to-guide
- Published: 2026-02-23

---

**DB-GPT supports fully offline deployments by routing LLM calls to local model servers like Ollama or llama.cpp through provider-specific TOML configuration files that map to client adapters in the `packages/dbgpt-core/src/dbgpt/model/` directory.**

DB-GPT enables complete air-gapped operation by connecting to locally hosted large language models instead of cloud APIs. This configuration leverages the provider-based plugin system implemented in the eosphoros-ai/DB-GPT repository, allowing you to switch between Ollama, llama.cpp, or vLLM without code changes. By editing the TOML configuration files and pointing to local endpoints, you can run DB-GPT entirely offline while maintaining full chat, RAG, and database interaction capabilities.

## Install and Start Your Local Model Server

Before configuring DB-GPT, you must have a local model server running. DB-GPT supports two primary patterns for local LLM access: HTTP-based proxy servers (Ollama) and direct model file loading (llama.cpp).

### Ollama Setup (HTTP Proxy Mode)

Ollama provides a lightweight HTTP API that serves quantized models without requiring complex GPU driver configurations. Install and start the server:

```bash

# Install Ollama (Linux/macOS)

curl -fsSL https://ollama.com/install.sh | sh

# Pull a model (e.g., deepseek-r1)

ollama pull deepseek-r1:1.5b

# Start the Ollama API server (default port 11434)

ollama serve &

```

The server listens on `http://localhost:11434` by default. DB-GPT's `OllamaLLMClient` (defined in [`packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py)) connects to this endpoint to route all completion requests.

### Llama.cpp Setup (Direct Model Loading)

For maximal performance with CPU-only or GPU-accelerated inference, use llama.cpp to load GGUF model files directly:

```bash

# Install the Python server helper

pip install llama-cpp-server-py

# Download a GGUF model and place it in your models directory

# Example: models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf

# Launch the server

python -m llama_cpp_server_py \
    --model models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf \
    --host 0.0.0.0 --port 8000 &

```

This exposes a compatible `/chat` endpoint that the `LlamaCppLLMClient` (located in [`packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py)) consumes.

## Configure DB-GPT for Local Backends

DB-GPT loads model definitions from TOML configuration files in the `configs/` directory. The system uses the `provider` field to determine which client class to instantiate via the registry in [`dbgpt/model/proxy/base.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/dbgpt/model/proxy/base.py).

### Ollama Proxy Configuration

Create or edit [`configs/dbgpt-proxy-ollama.toml`](https://github.com/eosphoros-ai/DB-GPT/blob/main/configs/dbgpt-proxy-ollama.toml) to point to your local Ollama instance:

```toml
[system]
language = "${env:DBGPT_LANG:-en}"
encrypt_key = "your_secret_key"

[service.web]
host = "0.0.0.0"
port = 5670

[rag.storage.vector]
type = "chroma"
persist_path = "pilot/data"

[models]
[[models.llms]]
name = "deepseek-r1:1.5b"
provider = "proxy/ollama"
api_base = "http://localhost:11434"
api_key = ""

```

The `provider = "proxy/ollama"` string triggers DB-GPT to instantiate `OllamaLLMClient`. The `api_base` parameter points to the local server; see lines 41-48 in [`ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/ollama.py) for the default dataclass parameters.

### Llama.cpp Local Configuration

For direct file-based loading without a separate server process, use the llama.cpp provider:

```toml
[[models.llms]]
name = "DeepSeek-R1-Distill-Qwen-1.5B"
provider = "llama.cpp"
path = "models/DeepSeek-R1-Distill-Qwen-1.5B-Q4_K_M.gguf"

```

When `provider = "llama.cpp"`, DB-GPT loads `LlamaCppLLMClient` and passes the `path` field directly to the underlying C++ library. The model path can be absolute or relative to the DB-GPT working directory.

## Launch and Verify the Offline Deployment

### Starting DB-GPT with Custom Config

Activate your virtual environment and launch the application with your chosen configuration:

```bash
source .venv/bin/activate

# For Ollama backend

python -m dbgpt_app.main --config configs/dbgpt-proxy-ollama.toml

# For llama.cpp backend

python -m dbgpt_app.main --config configs/dbgpt-local-llama-cpp.toml

```

The startup sequence performs the following operations:

1. Parses the TOML configuration via [`dbgpt/configs/model_config.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/dbgpt/configs/model_config.py)
2. Resolves the `provider` string to a client class using `register_proxy_model_adapter`
3. Instantiates the client through `OllamaLLMClient.new_client` (lines 82-94 in [`ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/ollama.py))
4. Forwards all chat and completion calls to the local server

### Testing the Local Connection

Verify your offline setup by programmatically calling the configured model:

```python
from dbgpt.core import ModelRequest
from dbgpt.model.proxy.base import ProxyModel

# Load the proxy model defined in the TOML config

proxy_model = ProxyModel.from_config_name("deepseek-r1:1.5b")
req = ModelRequest(prompt="Explain quantum entanglement in one sentence.")

# Stream the response from the local backend

for chunk in proxy_model.proxy_llm_client.sync_generate_stream(req):
    print(chunk.text, end="")

```

If text generates instantly without network latency, your offline backend is correctly wired.

## Switching Between Local Backends

DB-GPT's provider-based architecture allows backend switching without code modifications. To migrate from Ollama to llama.cpp (or vLLM, HuggingFace, etc.), simply:

1. Stop the DB-GPT service
2. Change the `provider` field in your TOML file (e.g., from `"proxy/ollama"` to `"llama.cpp"`)
3. Update backend-specific parameters (`api_base` for proxies, `path` for local files)
4. Restart with `python -m dbgpt_app.main --config your-new-config.toml`

The registry automatically loads the appropriate adapter class based on the provider string.

## Summary

- **Ollama proxy mode** uses HTTP connections to `localhost:11434` and requires the `provider = "proxy/ollama"` setting in your TOML file.
- **Direct local loading** via llama.cpp uses `provider = "llama.cpp"` with a file system `path` to your GGUF model.
- **Key source files** include [`packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/proxy/llms/ollama.py) for the Ollama client and [`packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/packages/dbgpt-core/src/dbgpt/model/llm/llama_cpp/llama_cpp.py) for the llama.cpp adapter.
- **Configuration files** are located in the `configs/` directory, with examples at [`configs/dbgpt-proxy-ollama.toml`](https://github.com/eosphoros-ai/DB-GPT/blob/main/configs/dbgpt-proxy-ollama.toml) and [`configs/dbgpt-local-llama-cpp.toml`](https://github.com/eosphoros-ai/DB-GPT/blob/main/configs/dbgpt-local-llama-cpp.toml).
- Launch DB-GPT offline using `python -m dbgpt_app.main --config <your-config.toml>` after starting your local model server.

## Frequently Asked Questions

### Can I run DB-GPT completely without internet access?

Yes. Once you have downloaded your model files (for llama.cpp) or pulled models via Ollama while online, DB-GPT operates entirely offline. The `api_base` for Ollama points to `localhost`, and the vector storage uses local ChromaDB (`type = "chroma"`) by default, ensuring no external API calls are required.

### What is the difference between "proxy/ollama" and "llama.cpp" providers?

The **"proxy/ollama"** provider uses `OllamaLLMClient` to send HTTP requests to a running Ollama server (typically on port 11434), similar to how cloud APIs work but locally. The **"llama.cpp"** provider uses `LlamaCppLLMClient` to load GGUF model files directly into memory via the llama-cpp-python bindings, bypassing network layers for lower latency.

### How do I configure a different local port for Ollama?

Modify the `api_base` parameter in your TOML configuration file. For example, if you started Ollama on port 8080, set `api_base = "http://localhost:8080"` in the `[[models.llms]]` section. The `OllamaLLMClient` reads this value during initialization (lines 82-94 in [`ollama.py`](https://github.com/eosphoros-ai/DB-GPT/blob/main/ollama.py)) and routes all requests to the specified endpoint.

### Does DB-GPT support other local backends like vLLM or HuggingFace?

Yes. The provider system supports multiple local inference engines. For vLLM, use `provider = "vllm"` with the appropriate `api_base` pointing to your vLLM server. For HuggingFace transformers local loading, use `provider = "hf"`. Each provider maps to a specific client class in `packages/dbgpt-core/src/dbgpt/model/`, following the same TOML configuration pattern as Ollama and llama.cpp.