How to Configure Multiple LLM Providers in Embabel: OpenAI, Anthropic, and Ollama

You can configure multiple LLM providers in Embabel by including provider-specific starter dependencies, mapping symbolic model names to concrete providers under embabel.models.llms, and injecting specific LlmService beans using Spring's @Qualifier annotation or the LlmServiceRegistry.

The embabel/embabel-agent repository provides a provider-agnostic abstraction layer that unifies disparate LLM vendors behind a common interface. By leveraging Spring Boot auto-configuration, you can simultaneously integrate OpenAI, Anthropic, Ollama, and OCI GenAI within the same application, switching between them via configuration or programmatic selection.

Understanding the Provider-Agnostic Architecture

Embabel’s architecture centers on the LlmService interface, which abstracts provider-specific implementations. Each LLM vendor ships an auto-configuration class (e.g., AgentOpenAiAutoConfiguration) that registers beans implementing this interface.

Key components include:

• LLM Model Registry: The property namespace embabel.models.llms maps symbolic names (like best or cheap) to provider-specific model identifiers (like gpt-4o or claude-3.5-sonnet).
• Default LLM: The property embabel.models.default-llm specifies which symbolic name to use when no explicit qualifier is provided.
• LlmServiceRegistry: A programmatic registry that exposes a get(String modelKey) method to fetch the appropriate service bean at runtime.

Step-by-Step Configuration Guide

1. Add Provider Dependencies

Include the starter modules for each LLM provider you need in your pom.xml. Each starter pulls in the necessary auto-configuration classes.

<!-- OpenAI -->
<dependency>
    <groupId>com.embabel</groupId>
    <artifactId>embabel-agent-starter-openai</artifactId>
    <version>${embabel.version}</version>
</dependency>

<!-- Anthropic -->
<dependency>
    <groupId>com.embabel</groupId>
    <artifactId>embabel-agent-starter-anthropic</artifactId>
    <version>${embabel.version}</version>
</dependency>

<!-- Ollama -->
<dependency>
    <groupId>com.embabel</groupId>
    <artifactId>embabel-agent-starter-ollama</artifactId>
    <version>${embabel.version}</version>
</dependency>

2. Configure Model Mappings

Define your model aliases and default provider in application.yml. Each entry under embabel.models.llms.<name> corresponds to a provider-specific model identifier.

embabel:
  models:
    default-llm: best
    llms:
      best: gpt-4o
      cheap: gpt-4o-mini
      anthropic: claude-3.5-sonnet
      ollama: llama3-70b-instruct

3. Inject Specific LLM Beans

To use multiple providers simultaneously within the same component, inject the desired LlmService bean using the @Qualifier annotation. The qualifier value must match the symbolic name defined in your configuration.

@Component
public class MultiLlmProcessor {

    private final LlmService openAiService;
    private final LlmService anthropicService;

    public MultiLlmProcessor(
            @Qualifier("best") LlmService openAiService,
            @Qualifier("anthropic") LlmService anthropicService) {
        this.openAiService = openAiService;
        this.anthropicService = anthropicService;
    }

    public Mono<String> answerWithOpenAi(String prompt) {
        return openAiService.createObject(prompt, String.class);
    }

    public Mono<String> answerWithAnthropic(String prompt) {
        return anthropicService.createObject(prompt, String.class);
    }
}

4. Dynamic Provider Selection at Runtime

For scenarios requiring runtime provider selection, use the LlmServiceRegistry to fetch beans dynamically by model key.

@Component
public class ChatOrchestrator {

    private final LlmServiceRegistry registry;

    public ChatOrchestrator(LlmServiceRegistry registry) {
        this.registry = registry;
    }

    public Mono<String> chat(String providerKey, String prompt) {
        LlmService service = registry.get(providerKey);
        return service.createObject(prompt, String.class);
    }
}

Provider-Specific Auto-Configuration Classes

Each provider is activated by a distinct auto-configuration class located in the embabel-agent-autoconfigure module. These classes read the embabel.models.llms entries and instantiate the appropriate LlmService implementation.

Provider	Auto-Configuration Class	Source Path
OpenAI	AgentOpenAiAutoConfiguration	embabel-agent-autoconfigure/models/embabel-agent-openai-autoconfigure/src/main/java/com/embabel/agent/autoconfigure/models/openai/AgentOpenAiAutoConfiguration.java
Anthropic	AgentAnthropicAutoConfiguration	src/main/java/com/embabel/agent/autoconfigure/models/anthropic/AgentAnthropicAutoConfiguration.java
Ollama	AgentOllamaAutoConfiguration	src/main/java/com/embabel/agent/autoconfigure/models/ollama/AgentOllamaAutoConfiguration.java
OCI GenAI	OciGenAiEnvironmentPostProcessor	embabel-agent-autoconfigure/models/embabel-agent-oci-genai-autoconfigure/src/main/java/com/embabel/agent/autoconfigure/models/ocigenai/OciGenAiEnvironmentPostProcessor.java

Providers are only activated when their auto-configuration class is present on the classpath, allowing you to control availability via dependencies alone.

Provider-Specific Properties

Configure API keys, timeouts, and model parameters under provider-specific prefixes. These properties are handled automatically by the respective auto-configuration classes.

embabel:
  openai:
    api-key: ${OPENAI_API_KEY}
    timeout: 30s
  
  anthropic:
    api-key: ${ANTHROPIC_API_KEY}
    temperature: 0.7
  
  ollama:
    base-url: http://localhost:11434

Observability and Tracing

When you configure multiple LLM providers, the observation subsystem automatically tags spans with embabel.llm.model based on the bean name. This enables unified metrics and tracing across all providers. The relevant conventions are implemented in:

• EmbabelLlmObservationConvention (embabel-agent-observability/src/main/java/com/embabel/agent/observability/tracing/EmbabelLlmObservationConvention.java)
• EmbabelSpanEventListener (embabel-agent-observability/src/main/java/com/embabel/agent/observability/tracing/EmbabelSpanEventListener.java)

Summary

• Add starter dependencies for each provider you want to use (OpenAI, Anthropic, Ollama, etc.).
• Map symbolic names to concrete model IDs under embabel.models.llms in your YAML configuration.
• Set a default provider using embabel.models.default-llm for fallback injection.
• Inject specific beans using @Qualifier with the symbolic model name, or use LlmServiceRegistry for dynamic lookup.
• Configure provider-specific settings (API keys, timeouts) under dedicated property namespaces like embabel.openai or embabel.anthropic.

Frequently Asked Questions

How do I add a custom LLM provider to Embabel?

You can add a custom provider by creating a new auto-configuration class annotated with @Configuration that registers a bean implementing LlmService. Name the bean after the model identifier and place the class in your classpath. The AgentZaiAutoConfiguration class in the source code serves as an example of this pattern for the Z-AI provider.

Can I use the same model name for different providers?

No, the symbolic names under embabel.models.llms must be unique as they serve as bean qualifiers. If you need to reference the same conceptual model across providers, use distinct keys (e.g., gpt-best and claude-best) and map them accordingly in your configuration.

How does Embabel handle API keys for different providers?

Each provider's auto-configuration class reads API keys from its dedicated property namespace. For example, AgentOpenAiAutoConfiguration consumes embabel.openai.api-key, while AgentAnthropicAutoConfiguration consumes embabel.anthropic.api-key. These are resolved independently, allowing you to securely manage credentials for multiple providers simultaneously.

What happens if the default LLM provider is unavailable?

If the default LLM provider specified in embabel.models.default-llm is unavailable, the application will throw a standard Spring NoSuchBeanDefinitionException at runtime if no fallback is configured. To handle provider failures gracefully, implement circuit breakers or use the LlmServiceRegistry to programmatically fall back to an alternative provider when a specific bean is unavailable.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}