# How to Configure RAG with ODS: Enable Vector Search and Embeddings

> Configure RAG with ODS for powerful vector search and embeddings. Easily install with the --rag flag and customize integrations via environment variables in your .env file.

- Repository: [Osmantic/ODS](https://github.com/Osmantic/ODS)
- Tags: how-to-guide
- Published: 2026-09-01

---

**Enable RAG in ODS by passing the `--rag` flag during installation, which automatically configures Qdrant for vector storage and Text-Embeddings-Inference (TEI) for embeddings, then customize the integration through environment variables in the `.env` file.**

ODS (Open Data Services) ships with a built-in **Retrieval-Augmented Generation (RAG)** stack that combines Qdrant and Text-Embeddings-Inference to enhance Open-WebUI with document retrieval capabilities. Understanding how to configure RAG with ODS requires navigating the installer's feature flags and the environment variables defined in the Docker Compose configuration.

## Understanding the ODS RAG Architecture

The RAG stack in ODS consists of two core services orchestrated through the installer. **Qdrant** serves as the vector database for storing document embeddings, while **Text-Embeddings-Inference (TEI)** provides the embedding engine. According to the source code in [`ods/installers/phases/03-features.sh`](https://github.com/Osmantic/ODS/blob/main/ods/installers/phases/03-features.sh), the installer controls these services through three hierarchical flags:

- **`ENABLE_RAG`** — The master switch set via the `--rag` CLI flag or interactive prompt (lines 61-66)
- **`ENABLE_QDRANT`** — Controls the vector store service, defaulting to `${ENABLE_RAG:-false}` (lines 32-35)
- **`ENABLE_EMBEDDINGS`** — Controls the TEI service, defaulting to `${ENABLE_RAG:-false}` (lines 32-35)

When `ENABLE_RAG=true`, the installer invokes `_sync_extension_compose` to activate the corresponding compose files located in `ods/extensions/services/qdrant/` and `ods/extensions/services/embeddings/` (lines 72-78). If disabled, these files are renamed with a `.disabled` suffix, ensuring [`scripts/resolve-compose-stack.sh`](https://github.com/Osmantic/ODS/blob/main/scripts/resolve-compose-stack.sh) excludes them from the Docker Compose stack.

## Enabling RAG During Installation

You can activate the RAG stack during the initial setup or subsequent reconfigurations.

**CLI Method**

Pass the `--rag` flag to the installer to automatically enable both Qdrant and TEI:

```bash
curl -sSf https://install.ods.ai | bash -s -- --rag

```

**Interactive Method**

During Phase 3 of the installation, answer "yes" to the RAG prompt. The installer sets `ENABLE_RAG=true` and propagates this value to the dependent Qdrant and Embeddings flags.

## Configuring Embedding Providers and Models

The Open-WebUI integration relies on environment variables injected through [`ods/docker-compose.base.yml`](https://github.com/Osmantic/ODS/blob/main/ods/docker-compose.base.yml) (lines 123-130). By default, ODS uses the bundled TEI service, but you can redirect to any OpenAI-compatible embedding endpoint.

**Default Configuration (Bundled TEI)**

```yaml
RAG_EMBEDDING_ENGINE: "${RAG_EMBEDDING_ENGINE:-openai}"
RAG_EMBEDDING_MODEL: "${RAG_EMBEDDING_MODEL:-${EMBEDDING_MODEL:-BAAI/bge-base-en-v1.5}}"
RAG_OPENAI_API_BASE_URL: "${RAG_OPENAI_API_BASE_URL:-http://embeddings:80/v1}"
RAG_OPENAI_API_KEY: "${RAG_OPENAI_API_KEY:-}"

```

**Key Variables:**

- **`RAG_EMBEDDING_MODEL`** — Defaults to `BAAI/bge-base-en-v1.5` for the bundled TEI service. When using external providers, set this to the provider-specific model identifier.
- **`RAG_OPENAI_API_BASE_URL`** — Points to the embeddings API. The default `http://embeddings:80/v1` targets the internal TEI container.
- **`RAG_OPENAI_API_KEY`** — Required for authenticated external endpoints. The installer validates this in [`scripts/validate-env.sh`](https://github.com/Osmantic/ODS/blob/main/scripts/validate-env.sh) (lines 494-509) when a custom base URL is supplied.

## Post-Installation Configuration

To modify RAG settings after installation, edit the `.env` file generated in your ODS installation directory. The installer preserves these values across re-runs, as verified in [`tests/smoke/installer-env-smoke.sh`](https://github.com/Osmantic/ODS/blob/main/tests/smoke/installer-env-smoke.sh) (lines 216-222).

**Switching to an External Provider:**

```dotenv
RAG_OPENAI_API_BASE_URL=https://my-embeddings.example.com/v1
RAG_EMBEDDING_MODEL=external-model-v2
RAG_OPENAI_API_KEY=your-secret-key

```

Apply changes by restarting the stack:

```bash
docker compose down
docker compose up -d

```

## Verifying the RAG Configuration

Confirm that both Qdrant and the embeddings service are operational using the built-in health checks.

**Check Qdrant Health**

The installer probes Qdrant during Phase 12 ([`installers/phases/12-health.sh`](https://github.com/Osmantic/ODS/blob/main/installers/phases/12-health.sh), lines 371-617). Manually verify with:

```bash
curl -s http://127.0.0.1:6333/health | jq .

```

**Check Open-WebUI Connectivity**

Test the integration endpoint to ensure Open-WebUI can reach the embeddings service:

```bash
curl -sf http://localhost:3000/api/test/rag

```

A successful response indicates the RAG pipeline is ready for document ingestion.

## ARM64 and Platform-Specific Limitations

On ARM64/aarch64 hosts, the installer automatically disables Qdrant and TEI due to upstream image compatibility issues (amd64-only architectures or page-size incompatibilities). This logic in [`installers/phases/03-features.sh`](https://github.com/Osmantic/ODS/blob/main/installers/phases/03-features.sh) (lines 46-60) forces `ENABLE_QDRANT=false` and `ENABLE_EMBEDDINGS=false` regardless of the `--rag` flag.

If you have compatible ARM64 images, manually override by setting `ENABLE_QDRANT=true` and `ENABLE_EMBEDDINGS=true` in `.env` and running a fresh installation.

## Summary

- **Enable RAG** using the `--rag` CLI flag or interactive prompt, which sets `ENABLE_RAG=true` and activates Qdrant and TEI via `_sync_extension_compose` in [`03-features.sh`](https://github.com/Osmantic/ODS/blob/main/03-features.sh).
- **Customize providers** by editing environment variables in `.env`, specifically `RAG_OPENAI_API_BASE_URL` and `RAG_EMBEDDING_MODEL`, with validation handled in [`validate-env.sh`](https://github.com/Osmantic/ODS/blob/main/validate-env.sh).
- **Verify deployment** using the Qdrant health endpoint on port 6333 and the Open-WebUI `/api/test/rag` endpoint.
- **Note ARM64 limitations** where RAG services are disabled by default due to image architecture constraints.

## Frequently Asked Questions

### What is the default embedding model used when I configure RAG with ODS?

ODS defaults to the `BAAI/bge-base-en-v1.5` model served by the bundled Text-Embeddings-Inference (TEI) container. This is defined in [`docker-compose.base.yml`](https://github.com/Osmantic/ODS/blob/main/docker-compose.base.yml) where `RAG_EMBEDDING_MODEL` inherits from the `EMBEDDING_MODEL` variable, falling back to this HuggingFace model.

### Can I use an external OpenAI-compatible embedding API instead of the bundled TEI service?

Yes. Set `RAG_OPENAI_API_BASE_URL` to your provider's endpoint (e.g., `https://api.openai.com/v1`) and update `RAG_EMBEDDING_MODEL` to match the provider's model name. If authentication is required, provide `RAG_OPENAI_API_KEY`. The installer validates these configurations in [`scripts/validate-env.sh`](https://github.com/Osmantic/ODS/blob/main/scripts/validate-env.sh).

### Why is RAG disabled on my ARM64/aarch64 server?

The installer disables Qdrant and TEI on ARM64 hosts because upstream container images are typically built for amd64 architectures or exhibit page-size incompatibilities. This safeguard is implemented in [`installers/phases/03-features.sh`](https://github.com/Osmantic/ODS/blob/main/installers/phases/03-features.sh) (lines 46-60). You can manually enable the services if you supply compatible ARM64 images.

### How do I verify that RAG is properly configured and running?

Check Qdrant's health endpoint on port 6333 using `curl -s http://127.0.0.1:6333/health`. Additionally, query the Open-WebUI test endpoint with `curl -sf http://localhost:3000/api/test/rag`—a successful response confirms the embeddings service is reachable from the web interface.