How to Implement a Custom EmbeddingModelFactory for a New Embedder in DAT

To add a custom embedder to DAT, implement the EmbeddingModelFactory SPI from ai.dat.core.factories, define typed configuration keys, and register your factory via Java's ServiceLoader mechanism.

DAT (Data AI Transformer) treats every embedding provider as a pluggable factory that conforms to the EmbeddingModelFactory interface. According to the junjiem/dat source code, this architecture allows you to integrate any embedding backend—whether commercial, open-source, or self-hosted—by supplying a factory class and a service registration file.

Understanding the EmbeddingModelFactory SPI

The contract for embedding integration lives in dat-core/src/main/java/ai/dat/core/factories/EmbeddingModelFactory.java. A valid implementation must supply four key pieces:

• Identifier: A string constant used in YAML configuration (e.g., "myembedder")
• Configuration Options: Typed keys for requiredOptions(), optionalOptions(), and fingerprintOptions() that define valid configuration parameters
• Factory Method: The create(ReadableConfig) method that instantiates the actual dev.langchain4j.model.embedding.EmbeddingModel
• Service Registration: A file in META-INF/services/ that enables Java's ServiceLoader to discover your factory at runtime

The existing XinferenceEmbeddingModelFactory and OpenAiEmbeddingModelFactory in the dat-embedders module provide canonical reference implementations for this pattern.

Step-by-Step Implementation Guide

Create a New Embedder Module

Start by creating a Maven module under the dat-embedders directory following the convention dat-embedder-<name>. The module structure mirrors dat-embedder-xinference, which contains:

dat-embedder-myembedder/
├── src/main/java/ai/dat/embedder/myembedder/
│   └── MyEmbeddingModelFactory.java
├── src/main/resources/META-INF/services/
│   └── ai.dat.core.factories.EmbeddingModelFactory
└── pom.xml

Add your embedder-specific dependency to the pom.xml. For example:

<dependency>
    <groupId>com.example</groupId>
    <artifactId>my-embedder</artifactId>
    <version>1.0</version>
</dependency>

Define Configuration Options

Configuration options use DAT's type-safe ConfigOption API. Define public static final constants for every parameter your embedder requires:

public static final ConfigOption<String> BASE_URL =
    ConfigOptions.key("base-url")
        .stringType()
        .noDefaultValue()
        .withDescription("Base URL of the MyEmbedder service.");

public static final ConfigOption<String> API_KEY =
    ConfigOptions.key("api-key")
        .stringType()
        .noDefaultValue()
        .withDescription("API key for MyEmbedder.");

public static final ConfigOption<Duration> TIMEOUT =
    ConfigOptions.key("timeout")
        .durationType()
        .noDefaultValue()
        .withDescription("Maximum request duration.");

Implement the Factory Interface

Your factory must implement EmbeddingModelFactory and override five methods. Here is the minimal implementation structure:

package ai.dat.embedder.myembedder;

import ai.dat.core.configuration.*;
import ai.dat.core.factories.*;
import ai.dat.core.utils.FactoryUtil;
import com.google.common.base.Preconditions;
import dev.langchain4j.model.embedding.EmbeddingModel;
import com.example.myembedder.MyEmbeddingModel;

import java.time.Duration;
import java.util.*;

public class MyEmbeddingModelFactory implements EmbeddingModelFactory {

    public static final String IDENTIFIER = "myembedder";

    // Configuration keys defined above...

    @Override
    public String factoryIdentifier() { 
        return IDENTIFIER; 
    }

    @Override
    public Set<ConfigOption<?>> requiredOptions() {
        return new LinkedHashSet<>(List.of(BASE_URL, API_KEY));
    }

    @Override
    public Set<ConfigOption<?>> optionalOptions() {
        return new LinkedHashSet<>(List.of(TIMEOUT));
    }

    @Override
    public Set<ConfigOption<?>> fingerprintOptions() {
        return Set.of(BASE_URL, API_KEY);
    }

    @Override
    public EmbeddingModel create(ReadableConfig config) {
        FactoryUtil.validateFactoryOptions(this, config);
        
        String baseUrl = config.get(BASE_URL);
        String apiKey = config.get(API_KEY);
        
        MyEmbeddingModel.MyEmbeddingModelBuilder builder = 
            MyEmbeddingModel.builder()
                .baseUrl(baseUrl)
                .apiKey(apiKey);

        config.getOptional(TIMEOUT).ifPresent(builder::timeout);
        return builder.build();
    }
}

The fingerprintOptions() set determines which configuration keys uniquely identify the model instance for caching purposes, while FactoryUtil.validateFactoryOptions() ensures all required options are present before instantiation.

Register with ServiceLoader

Create the service provider file at src/main/resources/META-INF/services/ai.dat.core.factories.EmbeddingModelFactory. This file must contain a single line with your factory's fully-qualified class name:

ai.dat.embedder.myembedder.MyEmbeddingModelFactory

This registration mechanism follows the same pattern as the Xinference embedder's service file located at dat-embedders/dat-embedder-xinference/src/main/resources/META-INF/services/ai.dat.core.factories.EmbeddingModelFactory.

Configure in DAT Projects

Once deployed, reference your custom embedder in project.yaml using the identifier string:

embedder: myembedder
embedder-config:
  base-url: https://api.myembedder.com
  api-key: ${MYEMBEDDER_API_KEY}
  timeout: 30s

DAT's runtime loads the factory by matching the embedder value against factoryIdentifier() return values.

Summary

• Implement EmbeddingModelFactory: Override factoryIdentifier(), requiredOptions(), optionalOptions(), fingerprintOptions(), and create() to define your embedder contract.
• Use ConfigOption: Define typed, documented configuration keys for all parameters your embedder accepts.
• Validate with FactoryUtil: Call FactoryUtil.validateFactoryOptions(this, config) in create() to enforce required parameters.
• Register via ServiceLoader: Create the file META-INF/services/ai.dat.core.factories.EmbeddingModelFactory containing your class name.
• Follow Existing Patterns: Reference XinferenceEmbeddingModelFactory and OpenAiEmbeddingModelFactory in dat-embedders for production-ready examples of error handling and builder patterns.

Frequently Asked Questions

What is the purpose of fingerprintOptions() in an EmbeddingModelFactory?

The fingerprintOptions() method returns a set of ConfigOption keys that uniquely identify the embedding model instance. DAT uses this fingerprint for caching and deduplication purposes, ensuring that identical configurations reuse the same model instance across different pipeline stages.

How do I make configuration parameters optional in my custom embedder?

Add the ConfigOption to the set returned by optionalOptions() instead of requiredOptions(). In the create() method, use config.getOptional(OPTION_NAME) which returns a java.util.Optional, allowing you to provide default behavior when the parameter is absent.

Where should I place validation logic for custom embedder configurations?

Implement validation in the create(ReadableConfig config) method after calling FactoryUtil.validateFactoryOptions(). Use Preconditions.checkArgument() from Guava or standard Java assertions to validate business rules, such as ensuring timeout values are non-negative or that URL strings are well-formed.

Can I expose a convenience factory method in the public API?

Yes. For user-friendly access, add a static factory method to DatEmbeddingModels (or your own utility class) that delegates to FactoryUtil.createEmbeddingModel("myembedder", config). This allows users to instantiate your embedder programmatically without manually loading the service provider.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}