How to Configure LiteLLM for Multi-Provider LLM Support in MathModelAgent

MathModelAgent routes all LLM requests through LiteLLM using environment-driven configuration, allowing each agent (Coordinator, Modeler, Coder, Writer) to call different providers like OpenAI, Anthropic, or Groq without code changes.

MathModelAgent abstracts every large language model interaction behind a unified interface powered by LiteLLM. This architecture enables different specialized agents to leverage distinct providers and models simultaneously, configured entirely through environment variables in the Settings module.

Architecture Overview: The Unified LLM Interface

In backend/app/core/llm/llm.py, the LLM class serves as the bridge between the application and various AI providers. It uses LiteLLM’s acompletion function to handle async requests, automatically routing calls based on the model string and credentials provided.

The system supports four distinct agent roles, each configurable independently:

• Coordinator: Manages workflow planning and task delegation
• Modeler: Handles mathematical modeling logic and analysis
• Coder: Generates, reviews, and debugs code
• Writer: Produces documentation and result summaries

Step 1 – Configure Provider Credentials in Settings

All provider-specific configuration resides in backend/app/config/setting.py. The Settings class uses Pydantic’s BaseSettings to load environment variables for each agent trio:

# backend/app/config/setting.py

class Settings(BaseSettings):
    COORDINATOR_API_KEY: Optional[str] = None
    COORDINATOR_MODEL: Optional[str] = None
    COORDINATOR_BASE_URL: Optional[str] = None
    
    MODELER_API_KEY: Optional[str] = None
    MODELER_MODEL: Optional[str] = None
    MODELER_BASE_URL: Optional[str] = None
    
    # Identical pattern for CODER and WRITER prefixes

Source: setting.py lines 18-38

Each agent requires exactly three fields:

• *_API_KEY: Authentication token for the provider
• *_MODEL: Model identifier (e.g., gpt-4o, claude-3-sonnet-20240229)
• *_BASE_URL: Optional custom endpoint for self-hosted or proxy APIs

Step 2 – Instantiate LLMs via the Factory Pattern

The LLMFactory in backend/app/core/llm/llm_factory.py bridges the configuration layer and the LiteLLM execution layer. It reads the Settings instance and constructs LLM objects for each agent:

# backend/app/core/llm/llm_factory.py

coordinator_llm = LLM(
    api_key=settings.COORDINATOR_API_KEY,
    model=settings.COORDINATOR_MODEL,
    base_url=settings.COORDINATOR_BASE_URL,
    task_id=self.task_id,
)

# Repeats for modeler, coder, and writer agents

Source: llm_factory.py lines 11-39

Calling factory.get_all_llms() returns a tuple of (coordinator_llm, modeler_llm, coder_llm, writer_llm), each pre-configured with their respective provider credentials and ready for async inference.

Step 3 – The LiteLLM Bridge Implementation

The LLM class implements the actual provider routing through LiteLLM’s async API. In backend/app/core/llm/llm.py, the chat() method constructs the request payload and invokes the completion:

# backend/app/core/llm/llm.py

kwargs = {
    "api_key": self.api_key,
    "model": self.model,
    "messages": history,
    "stream": False,
    "top_p": top_p,
    "metadata": {"agent_name": agent_name},
}
if self.base_url:
    kwargs["base_url"] = self.base_url

from litellm import acompletion
response = await acompletion(**kwargs)

Source: llm.py lines 52-70, 76-77

LiteLLM automatically detects the provider from the model string (e.g., gpt-4o routes to OpenAI, claude-3-sonnet routes to Anthropic) and handles the respective API endpoints, authentication headers, and payload formats transparently.

Observability and Callback Integration

MathModelAgent implements custom metrics collection via LiteLLM’s callback system. In backend/app/core/llm/llm.py, the agent_metrics callback is registered globally:

# backend/app/core/llm/llm.py

import litellm
litellm.callbacks = [agent_metrics]

Source: llm.py line 18

The callback implementation in backend/app/utils/track.py captures request metadata, token usage, and latency metrics, pushing these to Redis for real-time monitoring in the agent dashboard.

Source: track.py lines 1-2

Practical Configuration Examples

Sample .env.dev for Four Different Providers

Create a .env.dev file in the project root to assign different providers to each agent:

# Coordinator via OpenAI

COORDINATOR_API_KEY=sk-openai-xxxxxxxx
COORDINATOR_MODEL=gpt-4o
COORDINATOR_BASE_URL=https://api.openai.com/v1

# Modeler via Anthropic (base_url inferred by LiteLLM)

MODELER_API_KEY=sk-anthropic-xxxxxxxx
MODELER_MODEL=claude-3-sonnet-20240229

# Coder via OpenAI with cost-effective model

CODER_API_KEY=sk-openai-xxxxxxxx
CODER_MODEL=gpt-4o-mini
CODER_BASE_URL=https://api.openai.com/v1

# Writer via Groq for high-speed inference

WRITER_API_KEY=sk-groq-xxxxxxxx
WRITER_MODEL=llama3-70b-8192
WRITER_BASE_URL=https://api.groq.com/openai/v1

Runtime LLM Instantiation

Once environment variables are set, instantiate all agents in your task handler:

from app.core.llm.llm_factory import LLMFactory
from app.schemas.enums import AgentType

async def start_modeling_task(task_id: str):
    factory = LLMFactory(task_id)
    coordinator, modeler, coder, writer = factory.get_all_llms()
    
    # Coordinator plans the workflow

    plan = await coordinator.chat(
        history=[{"role": "user", "content": "Plan the optimization steps."}],
        agent_name=AgentType.COORDINATOR,
    )

Overriding Providers for Single Requests

For testing or edge cases, instantiate an LLM directly without the factory to bypass environment configuration:

from app.core.llm.llm import LLM
from app.schemas.enums import AgentType

# Route to local LLaMA server

local_llm = LLM(
    api_key="not-needed",
    model="llama3-8b",
    base_url="http://localhost:8000/v1",
    task_id="local-test",
)

response = await local_llm.chat(
    history=[{"role": "user", "content": "Solve this linear program."}],
    agent_name=AgentType.CODER,
)

Summary

• Environment-based configuration: Set *_API_KEY, *_MODEL, and optional *_BASE_URL variables in backend/app/config/setting.py via .env.dev files to switch providers instantly.
• Factory pattern: LLMFactory creates pre-configured LLM instances for each agent role, ensuring type safety and credential isolation.
• LiteLLM routing: The LLM.chat() method uses litellm.acompletion() to automatically route requests to OpenAI, Anthropic, Groq, or other supported providers based on the model identifier.
• Observability: Custom callbacks in backend/app/utils/track.py log all requests to Redis for real-time monitoring and cost tracking.
• Zero-downtime switching: Change providers by updating environment variables and restarting the backend—no code deployment required.

Frequently Asked Questions

Can I use the same provider for all four agents in MathModelAgent?

Yes. Set identical API keys and model names for all COORDINATOR_*, MODELER_*, CODER_*, and WRITER_* environment variables. The factory will create four LLM instances pointing to the same provider, though each maintains separate conversation history and metadata tracking via the task_id parameter.

How does LiteLLM know which provider endpoint to use?

LiteLLM parses the model parameter (e.g., gpt-4o, claude-3-sonnet-20240229, llama3-70b-8192) to determine the provider automatically. If using a custom endpoint or self-hosted model, provide the base_url parameter, which the LLM class injects into the acompletion call as shown in backend/app/core/llm/llm.py lines 68-69.

Where are the API credentials validated in the codebase?

Validation occurs at two levels: Pydantic validates that environment variables exist when Settings initializes, and LiteLLM validates the actual API key during the HTTP request to the provider endpoint. Invalid keys raise authentication errors at call time in backend/app/core/llm/llm.py during the await acompletion(**kwargs) execution.

Can I add custom callbacks beyond the agent_metrics logger?

Yes. Modify backend/app/core/llm/llm.py line 18 to append additional callback instances to the litellm.callbacks list. Each callback must implement LiteLLM’s callback interface (e.g., log_input_event, log_post_api_call). The existing agent_metrics callback in backend/app/utils/track.py serves as a reference implementation for Redis integration.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}