How to Configure RAG with ODS: Enable Vector Search and Embeddings

Enable RAG in ODS by passing the --rag flag during installation, which automatically configures Qdrant for vector storage and Text-Embeddings-Inference (TEI) for embeddings, then customize the integration through environment variables in the .env file.

ODS (Open Data Services) ships with a built-in Retrieval-Augmented Generation (RAG) stack that combines Qdrant and Text-Embeddings-Inference to enhance Open-WebUI with document retrieval capabilities. Understanding how to configure RAG with ODS requires navigating the installer's feature flags and the environment variables defined in the Docker Compose configuration.

Understanding the ODS RAG Architecture

The RAG stack in ODS consists of two core services orchestrated through the installer. Qdrant serves as the vector database for storing document embeddings, while Text-Embeddings-Inference (TEI) provides the embedding engine. According to the source code in ods/installers/phases/03-features.sh, the installer controls these services through three hierarchical flags:

• ENABLE_RAG — The master switch set via the --rag CLI flag or interactive prompt (lines 61-66)
• ENABLE_QDRANT — Controls the vector store service, defaulting to ${ENABLE_RAG:-false} (lines 32-35)
• ENABLE_EMBEDDINGS — Controls the TEI service, defaulting to ${ENABLE_RAG:-false} (lines 32-35)

When ENABLE_RAG=true, the installer invokes _sync_extension_compose to activate the corresponding compose files located in ods/extensions/services/qdrant/ and ods/extensions/services/embeddings/ (lines 72-78). If disabled, these files are renamed with a .disabled suffix, ensuring scripts/resolve-compose-stack.sh excludes them from the Docker Compose stack.

Enabling RAG During Installation

You can activate the RAG stack during the initial setup or subsequent reconfigurations.

CLI Method

Pass the --rag flag to the installer to automatically enable both Qdrant and TEI:

curl -sSf https://install.ods.ai | bash -s -- --rag

Interactive Method

During Phase 3 of the installation, answer "yes" to the RAG prompt. The installer sets ENABLE_RAG=true and propagates this value to the dependent Qdrant and Embeddings flags.

Configuring Embedding Providers and Models

The Open-WebUI integration relies on environment variables injected through ods/docker-compose.base.yml (lines 123-130). By default, ODS uses the bundled TEI service, but you can redirect to any OpenAI-compatible embedding endpoint.

Default Configuration (Bundled TEI)

RAG_EMBEDDING_ENGINE: "${RAG_EMBEDDING_ENGINE:-openai}"
RAG_EMBEDDING_MODEL: "${RAG_EMBEDDING_MODEL:-${EMBEDDING_MODEL:-BAAI/bge-base-en-v1.5}}"
RAG_OPENAI_API_BASE_URL: "${RAG_OPENAI_API_BASE_URL:-http://embeddings:80/v1}"
RAG_OPENAI_API_KEY: "${RAG_OPENAI_API_KEY:-}"

Key Variables:

• RAG_EMBEDDING_MODEL — Defaults to BAAI/bge-base-en-v1.5 for the bundled TEI service. When using external providers, set this to the provider-specific model identifier.
• RAG_OPENAI_API_BASE_URL — Points to the embeddings API. The default http://embeddings:80/v1 targets the internal TEI container.
• RAG_OPENAI_API_KEY — Required for authenticated external endpoints. The installer validates this in scripts/validate-env.sh (lines 494-509) when a custom base URL is supplied.

Post-Installation Configuration

To modify RAG settings after installation, edit the .env file generated in your ODS installation directory. The installer preserves these values across re-runs, as verified in tests/smoke/installer-env-smoke.sh (lines 216-222).

Switching to an External Provider:

RAG_OPENAI_API_BASE_URL=https://my-embeddings.example.com/v1
RAG_EMBEDDING_MODEL=external-model-v2
RAG_OPENAI_API_KEY=your-secret-key

Apply changes by restarting the stack:

docker compose down
docker compose up -d

Verifying the RAG Configuration

Confirm that both Qdrant and the embeddings service are operational using the built-in health checks.

Check Qdrant Health

The installer probes Qdrant during Phase 12 (installers/phases/12-health.sh, lines 371-617). Manually verify with:

curl -s http://127.0.0.1:6333/health | jq .

Check Open-WebUI Connectivity

Test the integration endpoint to ensure Open-WebUI can reach the embeddings service:

curl -sf http://localhost:3000/api/test/rag

A successful response indicates the RAG pipeline is ready for document ingestion.

ARM64 and Platform-Specific Limitations

On ARM64/aarch64 hosts, the installer automatically disables Qdrant and TEI due to upstream image compatibility issues (amd64-only architectures or page-size incompatibilities). This logic in installers/phases/03-features.sh (lines 46-60) forces ENABLE_QDRANT=false and ENABLE_EMBEDDINGS=false regardless of the --rag flag.

If you have compatible ARM64 images, manually override by setting ENABLE_QDRANT=true and ENABLE_EMBEDDINGS=true in .env and running a fresh installation.

Summary

• Enable RAG using the --rag CLI flag or interactive prompt, which sets ENABLE_RAG=true and activates Qdrant and TEI via _sync_extension_compose in 03-features.sh.
• Customize providers by editing environment variables in .env, specifically RAG_OPENAI_API_BASE_URL and RAG_EMBEDDING_MODEL, with validation handled in validate-env.sh.
• Verify deployment using the Qdrant health endpoint on port 6333 and the Open-WebUI /api/test/rag endpoint.
• Note ARM64 limitations where RAG services are disabled by default due to image architecture constraints.

Frequently Asked Questions

What is the default embedding model used when I configure RAG with ODS?

ODS defaults to the BAAI/bge-base-en-v1.5 model served by the bundled Text-Embeddings-Inference (TEI) container. This is defined in docker-compose.base.yml where RAG_EMBEDDING_MODEL inherits from the EMBEDDING_MODEL variable, falling back to this HuggingFace model.

Can I use an external OpenAI-compatible embedding API instead of the bundled TEI service?

Yes. Set RAG_OPENAI_API_BASE_URL to your provider's endpoint (e.g., https://api.openai.com/v1) and update RAG_EMBEDDING_MODEL to match the provider's model name. If authentication is required, provide RAG_OPENAI_API_KEY. The installer validates these configurations in scripts/validate-env.sh.

Why is RAG disabled on my ARM64/aarch64 server?

The installer disables Qdrant and TEI on ARM64 hosts because upstream container images are typically built for amd64 architectures or exhibit page-size incompatibilities. This safeguard is implemented in installers/phases/03-features.sh (lines 46-60). You can manually enable the services if you supply compatible ARM64 images.

How do I verify that RAG is properly configured and running?

Check Qdrant's health endpoint on port 6333 using curl -s http://127.0.0.1:6333/health. Additionally, query the Open-WebUI test endpoint with curl -sf http://localhost:3000/api/test/rag—a successful response confirms the embeddings service is reachable from the web interface.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}