How to Configure Multi-LLM Support in Embabel

Embabel enables multi-LLM support through a model-agnostic LlmOptions class that pairs hyper-parameters with pluggable ModelSelectionCriteria, allowing runtime routing by name, role, or fallback lists.

The embabel/embabel-agent repository provides a Spring Boot-based framework for building AI agents that can leverage multiple large language models simultaneously. By abstracting provider details behind a unified configuration layer, Embabel lets you define OpenAI, Anthropic, Azure, and other LLMs side-by-side, then route requests to the optimal model for each specific task.

Understanding Embabel's Multi-LLM Architecture

Embabel's architecture decouples LLM capability definition from provider implementation. The framework discovers provider-specific beans at startup and resolves them at runtime based on selection criteria you define.

The LlmOptions Class

At the center of multi-LLM configuration is LlmOptions, located in embabel-agent-common/embabel-agent-ai/src/main/kotlin/com/embabel/common/ai/model/LlmOptions.kt. This data class encapsulates:

• Hyper-parameters: Temperature, model name, API credentials
• Selection criteria: How the framework should choose the concrete provider
• Extensions: Provider-specific flags (e.g., Anthropic caching)

The criteria property (lines 38-52) computes the final ModelSelectionCriteria based on your configuration, determining which provider bean handles the request.

Model Selection Criteria

Embabel implements several concrete selection strategies in the ModelSelectionCriteria hierarchy:

• By Name: Explicitly select a configured model (e.g., "gpt-4")
• By Role: Map functional roles (e.g., "summarizer") to specific models
• Fallback-by-Name: Try models in priority order until one succeeds
• Platform Default: Use the globally configured default LLM

Provider modules register their beans via factories like OpenAiCompatibleModelFactory.kt in the embabel-agent-openai module, making them available to the selection engine.

Configuring Multiple LLM Providers in application.yml

Spring Boot's @ConfigurationProperties binds your YAML definitions to LlmOptions beans. Define each LLM under the embabel.ai.models prefix, specifying unique names, providers, and credentials.

embabel:
  ai:
    models:
      - name: openai-gpt4
        provider: openai
        apiKey: ${OPENAI_API_KEY}
        model: gpt-4
        temperature: 0.7
      - name: anthropic-claude
        provider: anthropic
        apiKey: ${ANTHROPIC_API_KEY}
        model: claude-2
        temperature: 0.6
    default: openai-gpt4
    roles:
      summarizer: anthropic-claude
      translator: openai-gpt4

Spring injects this list as distinct beans. The default key establishes the fallback when no specific criteria are provided, while the roles map enables semantic routing based on task type.

Selecting LLMs at Runtime

The static helper methods in LlmOptions (lines 44-95) construct the appropriate ModelSelectionCriteria without manual bean wiring.

By Explicit Name

Target a specific model configuration by its declared name:

import com.embabel.common.ai.model.LlmOptions

val claudeOpts = LlmOptions.withModel("anthropic-claude")

By Role

Route requests based on functional roles defined in your YAML:

val summarizerOpts = LlmOptions.withLlmForRole("summarizer")

Fallback Strategies

Define resilient routing that attempts multiple models in sequence:

val fallbackOpts = LlmOptions.withFirstAvailableLlmOf(
    "anthropic-claude", 
    "openai-gpt4"
)

Use the platform default when you want consistent behavior across the application:

val defaultOpts = LlmOptions.withDefaultLlm()

Using Options with AiService

Pass the configured options to AiService.chat() to execute against the resolved LLM:

val response = aiService.chat(
    prompt = "Summarize the following article...",
    options = LlmOptions.withLlmForRole("summarizer")
)
println(response)

Advanced Configuration and Provider Extensions

For provider-specific capabilities, use the extensions map in LlmOptions. Provider modules expose type-safe helpers to populate these values.

Enable Anthropic's caching extension (lines 92-104 in LlmOptions.kt):

val opts = LlmOptions.withModel("anthropic-claude")
    .withExtension("anthropicCaching", true)

Alternatively, use dedicated helpers if available:

val opts = LlmOptions.withModel("anthropic-claude")
    .withAnthropicCaching(true)

The extensions map allows the core framework to remain provider-agnostic while enabling access to unique features like thinking blocks, caching, or custom headers.

Summary

• Define LLMs in application.yml under embabel.ai.models with unique names and provider-specific credentials
• Register providers automatically via factories like OpenAiCompatibleModelFactory.kt using Spring's auto-configuration
• Select models programmatically using LlmOptions.withModel(), withLlmForRole(), or withFirstAvailableLlmOf()
• Resolve criteria at runtime through the criteria getter in LlmOptions.kt which maps to concrete provider beans
• Extend functionality via the extensions map for provider-specific features without breaking abstraction

Frequently Asked Questions

How does Embabel discover available LLM providers?

Provider modules in the embabel-agent repository, such as embabel-agent-openai, register their beans through factory classes like OpenAiCompatibleModelFactory.kt. Spring Boot's component scan picks up these factories during auto-configuration, making the models available to the ModelSelectionCriteria resolution engine without explicit bean declarations.

Can I use different API keys for different models?

Yes. Each entry under embabel.ai.models in your application.yml defines its own apiKey field. You can configure distinct keys for OpenAI, Anthropic, Azure, or DashScope within the same application, allowing you to mix commercial and enterprise endpoints with separate authentication.

What happens if the selected LLM is unavailable?

If you use LlmOptions.withFirstAvailableLlmOf(), Embabel attempts each model in the specified order until one succeeds. Without fallback configuration, the framework throws a resolution error if the criteria match an unavailable provider. Implementing fallback criteria is recommended for production resilience.

How do I add custom provider-specific options?

Use the withExtension() method on LlmOptions to populate the extensions map with arbitrary key-value pairs. Provider modules read these values to enable features like Anthropic's caching or custom reasoning parameters. This mechanism keeps your code portable while accessing provider-specific capabilities.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}