# How to Configure ODS to Use an External LLM Endpoint (Ollama or LM Studio)

> Configure ODS to use external LLMs like Ollama or LM Studio. Learn how ODS routes LLM traffic to OpenAI-compatible endpoints for seamless integration.

- Repository: [Osmantic/ODS](https://github.com/Osmantic/ODS)
- Tags: how-to-guide
- Published: 2026-08-30

---

**ODS routes all LLM traffic to external OpenAI-compatible endpoints by validating the host connection, rewriting the URL for container networking, and injecting a dedicated Docker Compose overlay that replaces local inference containers.**

The Osmantic/ODS repository allows you to replace its built-in inference backend—typically `llama-server` or Lemonade—with an external LLM service such as **Ollama** or **LM Studio**. This configuration leverages the standard OpenAI-compatible API to redirect model requests while maintaining the Dashboard API and Open WebUI functionality.

## Prerequisites for External LLM Configuration

Before running the installer, ensure your external LLM service is active and accessible on the host machine. Start Ollama or LM Studio and verify the HTTP API endpoint is responding.

For Ollama on Linux or macOS:

```bash
ollama serve &
curl http://127.0.0.1:11434/v1/models

```

Your service must expose the `/v1/models` endpoint for ODS to discover available models during installation.

## Configuring the External Endpoint via CLI Flags

The installer accepts explicit flags to define the external connection. According to [`install-core.sh`](https://github.com/Osmantic/ODS/blob/main/install-core.sh) (lines 161–169), the supported flags are:

- `--external-llm-url`: The host-facing URL (e.g., `http://127.0.0.1:11434`)
- `--external-llm-provider`: Either `ollama` or `lmstudio`
- `--external-llm-model`: The exact model name (e.g., `qwen3.5:9b`)
- `--reuse-external-llm`: Skip validation if the endpoint was previously configured
- `--no-external-llm`: Revert to managed local inference

Run the installation script with these parameters:

```bash
./install.sh \
  --external-llm-url http://127.0.0.1:11434 \
  --external-llm-provider ollama \
  --external-llm-model qwen3.5:9b \
  --non-interactive

```

The installer stores these values in `.env` and sets `LLM_BACKEND=external` for the Dashboard API service.

## Validation and Environment Variable Resolution

During phase [`02b-external-services.sh`](https://github.com/Osmantic/ODS/blob/main/02b-external-services.sh) (lines 106–116), the installer validates the connection by contacting `EXTERNAL_LLM_URL` and verifying the specified model exists in the remote catalog. If validation fails, the installation aborts immediately.

Upon success, the installer writes the following variables to `.env` (documented in `.env.example`, lines 100–110):

- `EXTERNAL_LLM_URL`: The original host-facing URL you provided
- `EXTERNAL_LLM_CONTAINER_URL`: Rewritten to `host.docker.internal` for internal container routing
- `EXTERNAL_LLM_PROVIDER`: Provider type determining discovery logic
- `EXTERNAL_LLM_MODEL`: The specific model identifier

All services use `LLM_API_BASE_PATH=/v1` and `ODS_TALK_VISION_URL="${EXTERNAL_LLM_CONTAINER_URL}/v1"` to communicate with the external endpoint.

## Understanding the Docker Compose Overlay

The [`docker-compose.external-llm.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.external-llm.yml) file functions as a composition overlay that removes local inference containers (`llama-server`, `model-router`) and injects `extra_hosts` mappings. As implemented in lines 24–30 of the overlay, this configuration ensures the Dashboard API and Open WebUI reach the external service via `host.docker.internal`.

The [`scripts/resolve-compose-stack.sh`](https://github.com/Osmantic/ODS/blob/main/scripts/resolve-compose-stack.sh) script automatically detects `EXTERNAL_LLM_URL` in your environment and appends the overlay to the final Docker Compose configuration. You can verify the merged configuration:

```bash
docker compose config | grep -A5 'dashboard-api'

```

Look for `extra_hosts` entries mapping `host.docker.internal` to `host-gateway` and the absence of `llama-server` services.

## Reverting to Local Inference

To restore the managed inference stack and remove external routing, execute the installer with the reversal flag:

```bash
./install.sh --no-external-llm

```

This clears `EXTERNAL_LLM_URL` from the environment and removes the [`docker-compose.external-llm.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.external-llm.yml) overlay, restoring the default `llama-server` or Lemonade containers in the next deployment.

## Summary

- **ODS** supports **Ollama** and **LM Studio** via OpenAI-compatible API endpoints by using a Docker Compose overlay that replaces local inference services.
- The installer validates external connectivity during phase [`02b-external-services.sh`](https://github.com/Osmantic/ODS/blob/main/02b-external-services.sh) and rewrites `EXTERNAL_LLM_URL` to `EXTERNAL_LLM_CONTAINER_URL` for internal Docker networking.
- Use CLI flags `--external-llm-url`, `--external-llm-provider`, and `--external-llm-model` to configure the connection during installation.
- The [`docker-compose.external-llm.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.external-llm.yml) overlay removes `llama-server` containers and adds `host.docker.internal` routing for the Dashboard API.
- Revert to local inference anytime by running `./install.sh --no-external-llm`.

## Frequently Asked Questions

### What URL format should I use for the external LLM endpoint?

Provide the base HTTP URL where Ollama or LM Studio listens on your host machine, typically `http://127.0.0.1:11434` for Ollama. The installer automatically rewrites this to `host.docker.internal:11434` in `EXTERNAL_LLM_CONTAINER_URL` so containers can reach the host service. Do not include `/v1` in the URL; ODS appends the API path internally using `LLM_API_BASE_PATH`.

### How does ODS validate that my external LLM is working?

During the [`02b-external-services.sh`](https://github.com/Osmantic/ODS/blob/main/02b-external-services.sh) installation phase, the installer attempts to connect to `EXTERNAL_LLM_URL` and queries the `/v1/models` endpoint to confirm the specific model you specified in `--external-llm-model` exists in the catalog. If the endpoint is unreachable or the model is missing, the installation halts with an error message before modifying your Docker Compose stack.

### Can I switch between external and local LLM backends after installation?

Yes. Run `./install.sh --no-external-llm` to disable external routing and restore the local `llama-server` containers. To switch back to external, rerun the installer with the `--external-llm-url` and related flags. The [`resolve-compose-stack.sh`](https://github.com/Osmantic/ODS/blob/main/resolve-compose-stack.sh) script dynamically includes or excludes the [`docker-compose.external-llm.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.external-llm.yml) overlay based on the presence of `EXTERNAL_LLM_URL` in your `.env` file.

### Why does the Dashboard API show `LLM_BACKEND=external`?

When an external endpoint is configured, the installer sets `LLM_BACKEND=external` in the Dashboard API environment variables (as seen in [`docker-compose.external-llm.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.external-llm.yml)). This signals the application to route all inference requests to `ODS_TALK_VISION_URL` instead of the local model router. The Dashboard UI hook in [`extensions/services/dashboard/src/hooks/useModels.js`](https://github.com/Osmantic/ODS/blob/main/extensions/services/dashboard/src/hooks/useModels.js) detects this state and displays an indicator that an external Ollama or LM Studio backend is active.