How to Use Custom Embedding Models with Different Backends in memU

Use HTTPEmbeddingClient with a specific provider argument to switch between OpenAI, Doubao, and other backends while using custom embedding models in the memU framework.

The NevaMind-AI/memU repository provides a flexible abstraction layer for integrating custom embedding models from various providers. By decoupling the HTTP transport logic from provider-specific payload formatting, memU allows you to swap between OpenAI, Doubao, and future backends without changing your application code.

Understanding the Backend Architecture for Custom Embedding Models

memU implements a backend → client pattern that isolates provider-specific logic into pluggable backend classes. The HTTPEmbeddingClient acts as the user-facing interface, while backend implementations handle the nuances of each embedding provider's API.

OpenAI Backend Implementation

The OpenAIEmbeddingBackend in src/memu/embedding/backends/openai.py constructs standard OpenAI-compatible requests. It builds a JSON payload with {"model": ..., "input": ...} and extracts the "embedding" field from the response array. This backend works with any provider offering an OpenAI-compatible HTTP interface.

Doubao Backend Implementation

The DoubaoEmbeddingBackend in src/memu/embedding/backends/doubao.py extends support for ByteDance's Doubao platform. It handles both text-only and multimodal embeddings, adding an encoding_format parameter and constructing complex payload structures for vision inputs. For multimodal calls, it builds lists of {type, ...} objects to accommodate mixed text, image, and video content.

Using HTTPEmbeddingClient with Custom Embedding Models

The HTTPEmbeddingClient in src/memu/embedding/http_client.py orchestrates the embedding process by selecting the appropriate backend at runtime based on the provider argument. Instantiate the client with your base_url, api_key, target embed_model, and provider identifier.

Text-Only Embeddings with OpenAI

To generate embeddings using OpenAI's API or any compatible endpoint:

import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def openai_example():
    client = HTTPEmbeddingClient(
        base_url="https://api.openai.com/v1",
        api_key="YOUR_OPENAI_KEY",
        embed_model="text-embedding-3-small",
        provider="openai"
    )
    texts = ["Hello world", "MemU makes RAG easy"]
    vectors = await client.embed(texts)
    print(vectors)

asyncio.run(openai_example())

Text-Only Embeddings with Doubao

For ByteDance's Doubao text embeddings:

import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def doubao_text_example():
    client = HTTPEmbeddingClient(
        base_url="https://ark.cn-beijing.volces.com",
        api_key="YOUR_DOUBAO_KEY",
        embed_model="doubao-embedding-base",
        provider="doubao"
    )
    texts = ["你好", "MemU 支持多模态"]
    vectors = await client.embed(texts)
    print(vectors)

asyncio.run(doubao_text_example())

Multimodal Embeddings with Doubao

Doubao supports vision inputs through the embed_multimodal method:

import asyncio
from memu.embedding.http_client import HTTPEmbeddingClient

async def doubao_multimodal_example():
    client = HTTPEmbeddingClient(
        base_url="https://ark.cn-beijing.volces.com",
        api_key="YOUR_DOUBAO_KEY",
        embed_model="doubao-embedding-vision-250615",
        provider="doubao"
    )
    inputs = [
        ("text", "Describe the image and video below."),
        ("image_url", "https://example.com/cat.png"),
        ("video_url", "https://example.com/clip.mp4")
    ]
    vectors = await client.embed_multimodal(inputs, encoding_format="float")
    print(vectors)

asyncio.run(doubao_multimodal_example())

Switching Models at Runtime

Change the embedding model without altering backend configuration by updating the embed_model parameter:

# Switch to a different OpenAI model

client = HTTPEmbeddingClient(
    base_url="https://api.openai.com/v1",
    api_key="YOUR_OPENAI_KEY",
    embed_model="text-embedding-3-large",
    provider="openai"
)
vectors = await client.embed(["sample text"])

Key Source Files for Custom Embedding Integration

File	Role
src/memu/embedding/http_client.py	Central HTTP client that selects the backend and makes embedding requests.
src/memu/embedding/backends/openai.py	Backend implementation for OpenAI‑compatible APIs.
src/memu/embedding/backends/doubao.py	Backend implementation for Doubao, including multimodal support.
src/memu/embedding/openai_sdk.py	Optional wrapper that uses the official OpenAI Python SDK.
src/memu/embedding/__init__.py	Exposes HTTPEmbeddingClient and OpenAIEmbeddingSDKClient for public use.

Summary

• Backend abstraction: memU uses pluggable backend classes (OpenAIEmbeddingBackend, DoubaoEmbeddingBackend) to isolate provider-specific payload formatting from transport logic.
• Unified client: HTTPEmbeddingClient in src/memu/embedding/http_client.py routes requests to the correct backend based on the provider argument.
• Multimodal support: Doubao backend supports text, image, and video inputs through embed_multimodal, while OpenAI backend handles standard text embeddings.
• Runtime flexibility: Switch embedding models or providers without code changes by updating embed_model and provider parameters.

Frequently Asked Questions

How do I add a new embedding provider to memU?

Create a new subclass of EmbeddingBackend in src/memu/embedding/backends/ implementing build_embedding_payload and parse_embedding_response methods. Register the backend in the EMBEDDING_BACKENDS mapping with a unique provider key. The HTTPEmbeddingClient will automatically route requests to your new backend when provider matches your registered key.

Can I use the official OpenAI SDK instead of the HTTP client?

Yes. Import OpenAIEmbeddingSDKClient from src/memu/embedding/openai_sdk.py to use the official OpenAI Python library. This client provides a convenience wrapper around the SDK but only supports OpenAI-compatible providers. For multi-provider setups or custom endpoints requiring specific payload formatting, use HTTPEmbeddingClient instead.

Does memU support multimodal embeddings for providers other than Doubao?

Currently, the multimodal embed_multimodal method is implemented specifically in DoubaoEmbeddingBackend within src/memu/embedding/backends/doubao.py. To add multimodal support for other providers, you would need to extend the base EmbeddingBackend class with multimodal payload building and response parsing methods, following the pattern established in the Doubao implementation.

How do I configure different embedding models for different tasks?

Instantiate separate HTTPEmbeddingClient instances with different embed_model parameters for each task, or reinitialize a single client with a new model ID when switching tasks. Since the client is lightweight and stateless aside from connection pooling, creating multiple clients for different models is the recommended approach for production applications requiring concurrent access to different embedding models.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}