# How to Use Custom Embedding Models with Different Backends in memU

> Dynamically use custom embedding models with OpenAI Doubao and more via HTTPEmbeddingClient in memU. Effortlessly switch backends for your AI projects.

- Repository: [NevaMind AI/memU](https://github.com/nevamind-ai/memu)
- Tags: how-to-guide
- Published: 2026-02-19

---

**Use `HTTPEmbeddingClient` with a specific `provider` argument to switch between OpenAI, Doubao, and other backends while using custom embedding models in the memU framework.**

The NevaMind-AI/memU repository provides a flexible abstraction layer for integrating custom embedding models from various providers. By decoupling the HTTP transport logic from provider-specific payload formatting, memU allows you to swap between OpenAI, Doubao, and future backends without changing your application code.

## Understanding the Backend Architecture for Custom Embedding Models

memU implements a **backend → client** pattern that isolates provider-specific logic into pluggable backend classes. The `HTTPEmbeddingClient` acts as the user-facing interface, while backend implementations handle the nuances of each embedding provider's API.

### OpenAI Backend Implementation

The `OpenAIEmbeddingBackend` in [`src/memu/embedding/backends/openai.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/backends/openai.py) constructs standard OpenAI-compatible requests. It builds a JSON payload with `{"model": ..., "input": ...}` and extracts the `"embedding"` field from the response array. This backend works with any provider offering an OpenAI-compatible HTTP interface.

### Doubao Backend Implementation

The `DoubaoEmbeddingBackend` in [`src/memu/embedding/backends/doubao.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/backends/doubao.py) extends support for ByteDance's Doubao platform. It handles both text-only and multimodal embeddings, adding an `encoding_format` parameter and constructing complex payload structures for vision inputs. For multimodal calls, it builds lists of `{type, ...}` objects to accommodate mixed text, image, and video content.

## Using HTTPEmbeddingClient with Custom Embedding Models

The `HTTPEmbeddingClient` in [`src/memu/embedding/http_client.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/http_client.py) orchestrates the embedding process by selecting the appropriate backend at runtime based on the `provider` argument. Instantiate the client with your `base_url`, `api_key`, target `embed_model`, and provider identifier.

### Text-Only Embeddings with OpenAI

To generate embeddings using OpenAI's API or any compatible endpoint:

```python
import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def openai_example():
    client = HTTPEmbeddingClient(
        base_url="https://api.openai.com/v1",
        api_key="YOUR_OPENAI_KEY",
        embed_model="text-embedding-3-small",
        provider="openai"
    )
    texts = ["Hello world", "MemU makes RAG easy"]
    vectors = await client.embed(texts)
    print(vectors)

asyncio.run(openai_example())

```

### Text-Only Embeddings with Doubao

For ByteDance's Doubao text embeddings:

```python
import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def doubao_text_example():
    client = HTTPEmbeddingClient(
        base_url="https://ark.cn-beijing.volces.com",
        api_key="YOUR_DOUBAO_KEY",
        embed_model="doubao-embedding-base",
        provider="doubao"
    )
    texts = ["你好", "MemU 支持多模态"]
    vectors = await client.embed(texts)
    print(vectors)

asyncio.run(doubao_text_example())

```

### Multimodal Embeddings with Doubao

Doubao supports vision inputs through the `embed_multimodal` method:

```python
import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def doubao_multimodal_example():
    client = HTTPEmbeddingClient(
        base_url="https://ark.cn-beijing.volces.com",
        api_key="YOUR_DOUBAO_KEY",
        embed_model="doubao-embedding-vision-250615",
        provider="doubao"
    )
    inputs = [
        ("text", "Describe the image and video below."),
        ("image_url", "https://example.com/cat.png"),
        ("video_url", "https://example.com/clip.mp4")
    ]
    vectors = await client.embed_multimodal(inputs, encoding_format="float")
    print(vectors)

asyncio.run(doubao_multimodal_example())

```

### Switching Models at Runtime

Change the embedding model without altering backend configuration by updating the `embed_model` parameter:

```python

# Switch to a different OpenAI model

client = HTTPEmbeddingClient(
    base_url="https://api.openai.com/v1",
    api_key="YOUR_OPENAI_KEY",
    embed_model="text-embedding-3-large",
    provider="openai"
)
vectors = await client.embed(["sample text"])

```

## Key Source Files for Custom Embedding Integration

| File | Role |
|------|------|
| [`src/memu/embedding/http_client.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/http_client.py) | Central HTTP client that selects the backend and makes embedding requests. |
| [`src/memu/embedding/backends/openai.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/backends/openai.py) | Backend implementation for OpenAI‑compatible APIs. |
| [`src/memu/embedding/backends/doubao.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/backends/doubao.py) | Backend implementation for Doubao, including multimodal support. |
| [`src/memu/embedding/openai_sdk.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/openai_sdk.py) | Optional wrapper that uses the official OpenAI Python SDK. |
| [`src/memu/embedding/__init__.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/__init__.py) | Exposes `HTTPEmbeddingClient` and `OpenAIEmbeddingSDKClient` for public use. |

## Summary

- **Backend abstraction**: memU uses pluggable backend classes (`OpenAIEmbeddingBackend`, `DoubaoEmbeddingBackend`) to isolate provider-specific payload formatting from transport logic.
- **Unified client**: `HTTPEmbeddingClient` in [`src/memu/embedding/http_client.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/http_client.py) routes requests to the correct backend based on the `provider` argument.
- **Multimodal support**: Doubao backend supports text, image, and video inputs through `embed_multimodal`, while OpenAI backend handles standard text embeddings.
- **Runtime flexibility**: Switch embedding models or providers without code changes by updating `embed_model` and `provider` parameters.

## Frequently Asked Questions

### How do I add a new embedding provider to memU?

Create a new subclass of `EmbeddingBackend` in `src/memu/embedding/backends/` implementing `build_embedding_payload` and `parse_embedding_response` methods. Register the backend in the `EMBEDDING_BACKENDS` mapping with a unique provider key. The `HTTPEmbeddingClient` will automatically route requests to your new backend when `provider` matches your registered key.

### Can I use the official OpenAI SDK instead of the HTTP client?

Yes. Import `OpenAIEmbeddingSDKClient` from [`src/memu/embedding/openai_sdk.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/openai_sdk.py) to use the official OpenAI Python library. This client provides a convenience wrapper around the SDK but only supports OpenAI-compatible providers. For multi-provider setups or custom endpoints requiring specific payload formatting, use `HTTPEmbeddingClient` instead.

### Does memU support multimodal embeddings for providers other than Doubao?

Currently, the multimodal `embed_multimodal` method is implemented specifically in `DoubaoEmbeddingBackend` within [`src/memu/embedding/backends/doubao.py`](https://github.com/NevaMind-AI/memU/blob/main/src/memu/embedding/backends/doubao.py). To add multimodal support for other providers, you would need to extend the base `EmbeddingBackend` class with multimodal payload building and response parsing methods, following the pattern established in the Doubao implementation.

### How do I configure different embedding models for different tasks?

Instantiate separate `HTTPEmbeddingClient` instances with different `embed_model` parameters for each task, or reinitialize a single client with a new model ID when switching tasks. Since the client is lightweight and stateless aside from connection pooling, creating multiple clients for different models is the recommended approach for production applications requiring concurrent access to different embedding models.