# How to Configure LiteLLM for Multi-Provider LLM Support in MathModelAgent

> Configure LiteLLM for multi-provider LLM support in MathModelAgent. Seamlessly switch between OpenAI, Anthropic, and Groq without code changes using environment variables.

- Repository: [Sanjin/mathmodelagent](https://github.com/jihe520/mathmodelagent)
- Tags: how-to-guide
- Published: 2026-03-04

---

**MathModelAgent routes all LLM requests through LiteLLM using environment-driven configuration, allowing each agent (Coordinator, Modeler, Coder, Writer) to call different providers like OpenAI, Anthropic, or Groq without code changes.**

MathModelAgent abstracts every large language model interaction behind a unified interface powered by LiteLLM. This architecture enables different specialized agents to leverage distinct providers and models simultaneously, configured entirely through environment variables in the Settings module.

## Architecture Overview: The Unified LLM Interface

In [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py), the `LLM` class serves as the bridge between the application and various AI providers. It uses LiteLLM’s `acompletion` function to handle async requests, automatically routing calls based on the `model` string and credentials provided.

The system supports four distinct agent roles, each configurable independently:

- **Coordinator**: Manages workflow planning and task delegation
- **Modeler**: Handles mathematical modeling logic and analysis
- **Coder**: Generates, reviews, and debugs code
- **Writer**: Produces documentation and result summaries

## Step 1 – Configure Provider Credentials in Settings

All provider-specific configuration resides in [`backend/app/config/setting.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/config/setting.py). The `Settings` class uses Pydantic’s `BaseSettings` to load environment variables for each agent trio:

```python

# backend/app/config/setting.py

class Settings(BaseSettings):
    COORDINATOR_API_KEY: Optional[str] = None
    COORDINATOR_MODEL: Optional[str] = None
    COORDINATOR_BASE_URL: Optional[str] = None
    
    MODELER_API_KEY: Optional[str] = None
    MODELER_MODEL: Optional[str] = None
    MODELER_BASE_URL: Optional[str] = None
    
    # Identical pattern for CODER and WRITER prefixes

```

*Source:* [setting.py lines 18-38](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/config/setting.py#L18-L38)

Each agent requires exactly three fields:

- `*_API_KEY`: Authentication token for the provider
- `*_MODEL`: Model identifier (e.g., `gpt-4o`, `claude-3-sonnet-20240229`)
- `*_BASE_URL`: Optional custom endpoint for self-hosted or proxy APIs

## Step 2 – Instantiate LLMs via the Factory Pattern

The `LLMFactory` in [`backend/app/core/llm/llm_factory.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm_factory.py) bridges the configuration layer and the LiteLLM execution layer. It reads the `Settings` instance and constructs `LLM` objects for each agent:

```python

# backend/app/core/llm/llm_factory.py

coordinator_llm = LLM(
    api_key=settings.COORDINATOR_API_KEY,
    model=settings.COORDINATOR_MODEL,
    base_url=settings.COORDINATOR_BASE_URL,
    task_id=self.task_id,
)

# Repeats for modeler, coder, and writer agents

```

*Source:* [llm_factory.py lines 11-39](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm_factory.py#L11-L39)

Calling `factory.get_all_llms()` returns a tuple of `(coordinator_llm, modeler_llm, coder_llm, writer_llm)`, each pre-configured with their respective provider credentials and ready for async inference.

## Step 3 – The LiteLLM Bridge Implementation

The `LLM` class implements the actual provider routing through LiteLLM’s async API. In [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py), the `chat()` method constructs the request payload and invokes the completion:

```python

# backend/app/core/llm/llm.py

kwargs = {
    "api_key": self.api_key,
    "model": self.model,
    "messages": history,
    "stream": False,
    "top_p": top_p,
    "metadata": {"agent_name": agent_name},
}
if self.base_url:
    kwargs["base_url"] = self.base_url

from litellm import acompletion
response = await acompletion(**kwargs)

```

*Source:* [llm.py lines 52-70, 76-77](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py#L52-L70%2C%20L76-L77)

LiteLLM automatically detects the provider from the `model` string (e.g., `gpt-4o` routes to OpenAI, `claude-3-sonnet` routes to Anthropic) and handles the respective API endpoints, authentication headers, and payload formats transparently.

## Observability and Callback Integration

MathModelAgent implements custom metrics collection via LiteLLM’s callback system. In [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py), the `agent_metrics` callback is registered globally:

```python

# backend/app/core/llm/llm.py

import litellm
litellm.callbacks = [agent_metrics]

```

*Source:* [llm.py line 18](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py#L18)

The callback implementation in [`backend/app/utils/track.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/utils/track.py) captures request metadata, token usage, and latency metrics, pushing these to Redis for real-time monitoring in the agent dashboard.

*Source:* [track.py lines 1-2](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/utils/track.py#L1-L2)

## Practical Configuration Examples

### Sample .env.dev for Four Different Providers

Create a `.env.dev` file in the project root to assign different providers to each agent:

```dotenv

# Coordinator via OpenAI

COORDINATOR_API_KEY=sk-openai-xxxxxxxx
COORDINATOR_MODEL=gpt-4o
COORDINATOR_BASE_URL=https://api.openai.com/v1

# Modeler via Anthropic (base_url inferred by LiteLLM)

MODELER_API_KEY=sk-anthropic-xxxxxxxx
MODELER_MODEL=claude-3-sonnet-20240229

# Coder via OpenAI with cost-effective model

CODER_API_KEY=sk-openai-xxxxxxxx
CODER_MODEL=gpt-4o-mini
CODER_BASE_URL=https://api.openai.com/v1

# Writer via Groq for high-speed inference

WRITER_API_KEY=sk-groq-xxxxxxxx
WRITER_MODEL=llama3-70b-8192
WRITER_BASE_URL=https://api.groq.com/openai/v1

```

### Runtime LLM Instantiation

Once environment variables are set, instantiate all agents in your task handler:

```python
from app.core.llm.llm_factory import LLMFactory
from app.schemas.enums import AgentType

async def start_modeling_task(task_id: str):
    factory = LLMFactory(task_id)
    coordinator, modeler, coder, writer = factory.get_all_llms()
    
    # Coordinator plans the workflow

    plan = await coordinator.chat(
        history=[{"role": "user", "content": "Plan the optimization steps."}],
        agent_name=AgentType.COORDINATOR,
    )

```

### Overriding Providers for Single Requests

For testing or edge cases, instantiate an `LLM` directly without the factory to bypass environment configuration:

```python
from app.core.llm.llm import LLM
from app.schemas.enums import AgentType

# Route to local LLaMA server

local_llm = LLM(
    api_key="not-needed",
    model="llama3-8b",
    base_url="http://localhost:8000/v1",
    task_id="local-test",
)

response = await local_llm.chat(
    history=[{"role": "user", "content": "Solve this linear program."}],
    agent_name=AgentType.CODER,
)

```

## Summary

- **Environment-based configuration**: Set `*_API_KEY`, `*_MODEL`, and optional `*_BASE_URL` variables in [`backend/app/config/setting.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/config/setting.py) via `.env.dev` files to switch providers instantly.
- **Factory pattern**: `LLMFactory` creates pre-configured `LLM` instances for each agent role, ensuring type safety and credential isolation.
- **LiteLLM routing**: The `LLM.chat()` method uses `litellm.acompletion()` to automatically route requests to OpenAI, Anthropic, Groq, or other supported providers based on the model identifier.
- **Observability**: Custom callbacks in [`backend/app/utils/track.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/utils/track.py) log all requests to Redis for real-time monitoring and cost tracking.
- **Zero-downtime switching**: Change providers by updating environment variables and restarting the backend—no code deployment required.

## Frequently Asked Questions

### Can I use the same provider for all four agents in MathModelAgent?

Yes. Set identical API keys and model names for all `COORDINATOR_*`, `MODELER_*`, `CODER_*`, and `WRITER_*` environment variables. The factory will create four `LLM` instances pointing to the same provider, though each maintains separate conversation history and metadata tracking via the `task_id` parameter.

### How does LiteLLM know which provider endpoint to use?

LiteLLM parses the `model` parameter (e.g., `gpt-4o`, `claude-3-sonnet-20240229`, `llama3-70b-8192`) to determine the provider automatically. If using a custom endpoint or self-hosted model, provide the `base_url` parameter, which the `LLM` class injects into the `acompletion` call as shown in [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py) lines 68-69.

### Where are the API credentials validated in the codebase?

Validation occurs at two levels: Pydantic validates that environment variables exist when `Settings` initializes, and LiteLLM validates the actual API key during the HTTP request to the provider endpoint. Invalid keys raise authentication errors at call time in [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py) during the `await acompletion(**kwargs)` execution.

### Can I add custom callbacks beyond the agent_metrics logger?

Yes. Modify [`backend/app/core/llm/llm.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/core/llm/llm.py) line 18 to append additional callback instances to the `litellm.callbacks` list. Each callback must implement LiteLLM’s callback interface (e.g., `log_input_event`, `log_post_api_call`). The existing `agent_metrics` callback in [`backend/app/utils/track.py`](https://github.com/jihe520/mathmodelagent/blob/main/backend/app/utils/track.py) serves as a reference implementation for Redis integration.