# Creating Model Routing with RouteLLM for Cost Optimization in Agents > Optimize agent costs with RouteLLM. Automatically route queries between GPT-4o-mini and Nebius Llama-3.1 based on complexity, saving money without sacrificing quality. - Repository: [Arindam Majumder /awesome-ai-apps](https://github.com/Arindam200/awesome-ai-apps) - Tags: how-to-guide - Published: 2026-05-06 --- **RouteLLM automatically routes user queries between a high-cost strong model (GPT-4o-mini) and a low-cost weak model (Nebius Llama-3.1) based on predicted complexity, reducing API spend while maintaining response quality.** The `simple_ai_agents/llm_router` demo in the [Arindam200/awesome-ai-apps](https://github.com/Arindam200/awesome-ai-apps) repository demonstrates a practical implementation of **creating model routing with RouteLLM for cost optimization**. By leveraging the `Controller` class from the `routellm` package, this agent architecture dynamically selects the most cost-effective model for each request without manual intervention. ## Architecture of the RouteLLM Routing System The implementation uses a dual-model strategy to balance performance and expenditure. A **Streamlit** front-end collects prompts and API keys, then delegates routing decisions to the RouteLLM `Controller`. ### Strong vs. Weak Model Configuration At initialization (lines 99‑120 in [`simple_ai_agents/llm_router/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/simple_ai_agents/llm_router/main.py)), the system defines two distinct endpoints: - **Strong model**: `gpt-4o-mini` via OpenAI for complex, high-stakes queries requiring maximum accuracy. - **Weak model**: `meta-llama/Meta-Llama-3.1-70B-Instruct` served through Nebius Token Factory for routine queries at significantly lower cost. The `Controller` evaluates each incoming prompt and routes it to the appropriate backend based on learned thresholds. ### Cost-Optimization Flow The request lifecycle follows this pattern: ```text User → Streamlit UI ↓ (API keys) RouteLLM Controller (router‑mf‑0.11593) ↙ ↘ Strong model (GPT‑4o‑mini) Weak model (Nebius Llama‑3.1) (high‑quality, higher cost) (cost‑effective, lower‑price) ↘ ↙ Chosen model returns response → UI shows model badge ``` By delegating routine or low-complexity queries to the Nebius Llama model, the system saves API spend while falling back to GPT-4o-mini only when necessary. ## Implementing the RouteLLM Controller The core logic resides in [`simple_ai_agents/llm_router/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/simple_ai_agents/llm_router/main.py), which handles client initialization, request execution, and response formatting. ### Initializing the Router The `get_routellm_client` function instantiates a cached `Controller` to avoid re-initialization overhead: ```python from routellm.controller import Controller import streamlit as st from dotenv import load_dotenv load_dotenv() # Loads NEBIUS_API_KEY and OPENAI_API_KEY @st.cache_resource def get_routellm_client(): """Initialize and cache the RouteLLM controller.""" client = Controller( routers=["mf"], # router-mf identifier strong_model="gpt-4o-mini", weak_model="meta-llama/Meta-Llama-3.1-70B-Instruct", ) return client ``` This configuration uses the `router-mf-0.11593` model identifier to trigger the routing decision engine. ### Handling Conversation History When processing user input (lines 156‑182), the application constructs the message array and sends it through the routed client: ```python client = get_routellm_client() messages = [{"role": m["role"], "content": m["content"]} for m in st.session_state.messages] response = client.chat.completions.create( model="router-mf-0.11593", messages=messages, ) # Normalize response format (handles both dict and OpenAI object types) if isinstance(response, dict): reply = response["choices"][0]["message"]["content"] model_used = response.get("model", "unknown") else: reply = response.choices[0].message.content model_used = getattr(response, "model", "unknown") ``` The code accommodates both the raw dictionary format returned by the RouteLLM SDK and standard OpenAI-compatible objects, ensuring robust error handling. ### Displaying Model Attribution To provide transparency on cost and performance (lines 184‑202), the UI renders a color-coded badge indicating which backend served the request: ```python badge_color = "#667eea" if "gpt" in model_used.lower() else "#764ba2" st.markdown( f"" f"🤖 {model_used}", unsafe_allow_html=True, ) ``` Blue badges indicate OpenAI models, while purple denotes Nebius-based responses. ## Standalone Python Implementation For non-Streamlit agents, the routing logic simplifies to: ```python from routellm.controller import Controller router = Controller( routers=["mf"], strong_model="gpt-4o-mini", weak_model="meta-llama/Meta-Llama-3.1-70B-Instruct", ) def ask_route_llm(messages): """Send history to RouteLLM and return content with model name.""" response = router.chat.completions.create( model="router-mf-0.11593", messages=messages, ) if isinstance(response, dict): content = response["choices"][0]["message"]["content"] model_name = response.get("model", "unknown") else: content = response.choices[0].message.content model_name = getattr(response, "model", "unknown") return content, model_name ``` This pattern can be integrated into any autonomous agent framework to enable dynamic model selection. ## Summary - **RouteLLM** uses a `Controller` class to automatically select between a **strong model** (GPT-4o-mini) and a **weak model** (Nebius Llama-3.1) based on query complexity. - The implementation in [`simple_ai_agents/llm_router/main.py`](https://github.com/Arindam200/awesome-ai-apps/blob/main/simple_ai_agents/llm_router/main.py) demonstrates production-ready patterns including **API key management**, **response caching** via `@st.cache_resource`, and **format normalization**. - **Cost savings** are achieved by routing routine queries to cheaper models while preserving high-quality responses for complex tasks. - The **model badge UI pattern** provides users transparency regarding which backend processed their request. ## Frequently Asked Questions ### How does RouteLLM decide which model to use for each query? RouteLLM's `Controller` employs a trained classifier (specified by the `routers=["mf"]` configuration) that evaluates the prompt's semantic features against learned thresholds. If the predicted complexity exceeds the threshold for the weak model, the request automatically routes to the strong model (GPT-4o-mini); otherwise, it processes via the cost-effective Nebius Llama endpoint. ### Can I use different models than GPT-4o-mini and Llama-3.1? Yes. The `Controller` accepts any OpenAI-compatible endpoint for the `strong_model` and `weak_model` parameters. You can configure alternatives like Claude 3.5 Sonnet, GPT-4o, or other open models hosted on Nebius, AWS Bedrock, or local inference servers, provided they expose a compatible chat completions API. ### How are API keys managed securely in this implementation? The demo uses `python-dotenv` to load `NEBIUS_API_KEY` and `OPENAI_API_KEY` from environment variables defined in a `.env` file. These credentials are never hardcoded; they are passed directly to the underlying SDKs used by the RouteLLM `Controller`. For production deployments, migrate to secret management systems like AWS Secrets Manager or Azure Key Vault. ### What happens if the RouteLLM router is unavailable? If the routing service fails to classify the request or if neither model endpoint responds, the `client.chat.completions.create` call raises an exception. The reference implementation wraps these calls in try-except blocks to capture initialization and runtime errors, displaying user-friendly messages in the Streamlit interface without exposing sensitive stack traces.