Creating Model Routing with RouteLLM for Cost Optimization in Agents

RouteLLM automatically routes user queries between a high-cost strong model (GPT-4o-mini) and a low-cost weak model (Nebius Llama-3.1) based on predicted complexity, reducing API spend while maintaining response quality.

The simple_ai_agents/llm_router demo in the Arindam200/awesome-ai-apps repository demonstrates a practical implementation of creating model routing with RouteLLM for cost optimization. By leveraging the Controller class from the routellm package, this agent architecture dynamically selects the most cost-effective model for each request without manual intervention.

Architecture of the RouteLLM Routing System

The implementation uses a dual-model strategy to balance performance and expenditure. A Streamlit front-end collects prompts and API keys, then delegates routing decisions to the RouteLLM Controller.

Strong vs. Weak Model Configuration

At initialization (lines 99‑120 in simple_ai_agents/llm_router/main.py), the system defines two distinct endpoints:

• Strong model: gpt-4o-mini via OpenAI for complex, high-stakes queries requiring maximum accuracy.
• Weak model: meta-llama/Meta-Llama-3.1-70B-Instruct served through Nebius Token Factory for routine queries at significantly lower cost.

The Controller evaluates each incoming prompt and routes it to the appropriate backend based on learned thresholds.

Cost-Optimization Flow

The request lifecycle follows this pattern:

User → Streamlit UI
     ↓ (API keys)
RouteLLM Controller (router‑mf‑0.11593)
     ↙                     ↘
Strong model (GPT‑4o‑mini)   Weak model (Nebius Llama‑3.1)
(high‑quality, higher cost) (cost‑effective, lower‑price)
     ↘                     ↙
Chosen model returns response → UI shows model badge

By delegating routine or low-complexity queries to the Nebius Llama model, the system saves API spend while falling back to GPT-4o-mini only when necessary.

Implementing the RouteLLM Controller

The core logic resides in simple_ai_agents/llm_router/main.py, which handles client initialization, request execution, and response formatting.

Initializing the Router

The get_routellm_client function instantiates a cached Controller to avoid re-initialization overhead:

from routellm.controller import Controller
import streamlit as st
from dotenv import load_dotenv

load_dotenv()  # Loads NEBIUS_API_KEY and OPENAI_API_KEY

@st.cache_resource
def get_routellm_client():
    """Initialize and cache the RouteLLM controller."""
    client = Controller(
        routers=["mf"],  # router-mf identifier

        strong_model="gpt-4o-mini",
        weak_model="meta-llama/Meta-Llama-3.1-70B-Instruct",
    )
    return client

This configuration uses the router-mf-0.11593 model identifier to trigger the routing decision engine.

Handling Conversation History

When processing user input (lines 156‑182), the application constructs the message array and sends it through the routed client:

client = get_routellm_client()
messages = [{"role": m["role"], "content": m["content"]} 
            for m in st.session_state.messages]

response = client.chat.completions.create(
    model="router-mf-0.11593",
    messages=messages,
)

# Normalize response format (handles both dict and OpenAI object types)

if isinstance(response, dict):
    reply = response["choices"][0]["message"]["content"]
    model_used = response.get("model", "unknown")
else:
    reply = response.choices[0].message.content
    model_used = getattr(response, "model", "unknown")

The code accommodates both the raw dictionary format returned by the RouteLLM SDK and standard OpenAI-compatible objects, ensuring robust error handling.

Displaying Model Attribution

To provide transparency on cost and performance (lines 184‑202), the UI renders a color-coded badge indicating which backend served the request:

badge_color = "#667eea" if "gpt" in model_used.lower() else "#764ba2"
st.markdown(
    f"<span style='background-color: {badge_color}; color: white; "
    f"padding: 4px 12px; border-radius: 12px; font-size: 0.8em;'>"
    f"🤖 {model_used}</span>",
    unsafe_allow_html=True,
)

Blue badges indicate OpenAI models, while purple denotes Nebius-based responses.

Standalone Python Implementation

For non-Streamlit agents, the routing logic simplifies to:

from routellm.controller import Controller

router = Controller(
    routers=["mf"],
    strong_model="gpt-4o-mini",
    weak_model="meta-llama/Meta-Llama-3.1-70B-Instruct",
)

def ask_route_llm(messages):
    """Send history to RouteLLM and return content with model name."""
    response = router.chat.completions.create(
        model="router-mf-0.11593",
        messages=messages,
    )
    
    if isinstance(response, dict):
        content = response["choices"][0]["message"]["content"]
        model_name = response.get("model", "unknown")
    else:
        content = response.choices[0].message.content
        model_name = getattr(response, "model", "unknown")
    
    return content, model_name

This pattern can be integrated into any autonomous agent framework to enable dynamic model selection.

Summary

• RouteLLM uses a Controller class to automatically select between a strong model (GPT-4o-mini) and a weak model (Nebius Llama-3.1) based on query complexity.
• The implementation in simple_ai_agents/llm_router/main.py demonstrates production-ready patterns including API key management, response caching via @st.cache_resource, and format normalization.
• Cost savings are achieved by routing routine queries to cheaper models while preserving high-quality responses for complex tasks.
• The model badge UI pattern provides users transparency regarding which backend processed their request.

Frequently Asked Questions

How does RouteLLM decide which model to use for each query?

RouteLLM's Controller employs a trained classifier (specified by the routers=["mf"] configuration) that evaluates the prompt's semantic features against learned thresholds. If the predicted complexity exceeds the threshold for the weak model, the request automatically routes to the strong model (GPT-4o-mini); otherwise, it processes via the cost-effective Nebius Llama endpoint.

Can I use different models than GPT-4o-mini and Llama-3.1?

Yes. The Controller accepts any OpenAI-compatible endpoint for the strong_model and weak_model parameters. You can configure alternatives like Claude 3.5 Sonnet, GPT-4o, or other open models hosted on Nebius, AWS Bedrock, or local inference servers, provided they expose a compatible chat completions API.

How are API keys managed securely in this implementation?

The demo uses python-dotenv to load NEBIUS_API_KEY and OPENAI_API_KEY from environment variables defined in a .env file. These credentials are never hardcoded; they are passed directly to the underlying SDKs used by the RouteLLM Controller. For production deployments, migrate to secret management systems like AWS Secrets Manager or Azure Key Vault.

What happens if the RouteLLM router is unavailable?

If the routing service fails to classify the request or if neither model endpoint responds, the client.chat.completions.create call raises an exception. The reference implementation wraps these calls in try-except blocks to capture initialization and runtime errors, displaying user-friendly messages in the Streamlit interface without exposing sensitive stack traces.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}