# How the HF Router Catalog Auto-Selects Providers for Different Models in ML Intern

> Discover how the HF router catalog auto-selects providers in ML Intern. Learn about routing policies and dynamic provider selection for optimized ML model deployment.

- Repository: [Hugging Face/ml-intern](https://github.com/huggingface/ml-intern)
- Tags: internals
- Published: 2026-04-24

---

**ML Intern uses the Hugging Face Inference Router to fetch a live catalog of available providers every five minutes, then applies user-specified routing policies (fastest, cheapest, preferred) to filter live providers while letting the router backend make the final selection.**

The `huggingface/ml-intern` repository implements an intelligent routing layer that automatically discovers which cloud providers can serve a given model. By leveraging the HF router catalog, the system eliminates manual provider configuration and ensures optimal model serving through real-time metadata caching and policy-based selection.

## Fetching and Caching the Router Catalog

The auto-selection process begins in [`agent/core/hf_router_catalog.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/hf_router_catalog.py), which communicates with the Hugging Face Inference Router's public endpoint at `https://router.huggingface.co/v1/models`. This endpoint returns a JSON list containing every available model along with **provider-level metadata**, including status, context length, pricing, and tool-call support.

The `_fetch_catalog()` function retrieves this data with a short HTTP timeout and stores it in a module-level cache (`_cache`) for 300 seconds (`_CACHE_TTL_SECONDS = 300`). This five-minute TTL guarantees fast, low-latency routing decisions for subsequent lookups. If the HTTP request fails, the system falls back to a stale cache when available; otherwise, it returns an empty catalog to prevent system crashes.

## Parsing Provider Metadata into Structured Objects

Once fetched, the raw catalog data undergoes transformation through the `_parse_entry()` function. This converts each JSON entry into a structured `ModelInfo` dataclass that contains a list of `ProviderInfo` objects.

Each `ProviderInfo` captures critical attributes:

- `provider`: The provider identifier
- `status`: Availability state (e.g., *live* or *down*)
- `context_length`: Maximum token context
- `input_price` and `output_price`: Cost per million tokens
- `supports_tools` and `supports_structured_output`: Capability flags

The `ModelInfo` class exposes helper properties such as `live_providers`, `max_context_length`, and `any_supports_tools`, enabling downstream components to quickly reason about model availability and capabilities without parsing raw JSON.

## Auto-Selection Logic in the Model Switcher

The user-facing routing logic resides in [`agent/core/model_switcher.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/model_switcher.py). When a user executes the `/model` command, the system invokes `_print_hf_routing_info()` to process the model identifier and optional routing tags.

The function accepts model strings in the format `org/model:tag`, where the tag can be `:fastest`, `:cheapest`, `:preferred`, or a specific provider name. These correspond to the three supported routing policies defined in `_ROUTING_POLICIES = {"fastest", "cheapest", "preferred"}`. When a policy tag is provided, the system validates that matching providers exist in `live_providers` and displays the selected policy.

When no tag is supplied, the default policy is **"auto (fastest)"**, which lists all available live providers. The actual provider selection occurs later when the Hugging Face router service processes the first inference request. The system also emits warnings when no live providers exist for a model or when none advertise tool-call support, giving users the opportunity to intervene before the probe call fails.

## Practical Implementation Examples

```python

# Example 1 – Retrieve a model’s provider information

from agent.core import hf_router_catalog as cat

model = "MiniMaxAI/MiniMax-M2.7"
info = cat.lookup(model)          # → ModelInfo or None

if info:
    for p in info.live_providers:
        print(f"{p.provider}: ${p.input_price}/{p.output_price} per M tok "
              f"{p.context_length or 'n/a'} ctx "
              f"{'tools' if p.supports_tools else 'no tools'}")
else:
    # Not in the catalog – suggest close matches

    print("Did you mean:", cat.fuzzy_suggest(model))

```

```python

# Example 2 – Show routing details as the REPL does

from agent.core import model_switcher as ms
from rich.console import Console

console = Console()
ms._print_hf_routing_info("MiniMaxAI/MiniMax-M2.7:fastest", console)

# Output includes:

#   routing: fastest

#   huggingface: $0.02/$0.06 per M tok, 4,096 ctx, tools

```

```python

# Example 3 – Pre‑warm the catalog at startup (called in agent/main.py)

import asyncio
from agent.core import hf_router_catalog

asyncio.create_task(asyncio.to_thread(hf_router_catalog.prewarm))

# The catalog is fetched once so the first `/model` lookup is instantaneous.

```

## Summary

- The HF router catalog is fetched from `https://router.huggingface.co/v1/models` every five minutes and cached in memory via `_cache` to ensure low-latency lookups.
- Provider metadata is structured into `ModelInfo` and `ProviderInfo` dataclasses exposing properties like `live_providers` and `max_context_length`.
- Users can specify routing policies (`fastest`, `cheapest`, `preferred`) via tags appended to model names, or rely on the default "auto (fastest)" behavior.
- ML Intern delegates the final provider selection to the Hugging Face router backend, which chooses the actual provider when processing inference requests.
- The system gracefully handles failures by falling back to stale cache data and warns users about unavailable providers or missing capability support.

## Frequently Asked Questions

### How often does ML Intern refresh the HF router catalog?

The catalog refreshes every five minutes. The `_fetch_catalog()` function in [`agent/core/hf_router_catalog.py`](https://github.com/huggingface/ml-intern/blob/main/agent/core/hf_router_catalog.py) implements a time-based cache with `_CACHE_TTL_SECONDS = 300` that automatically expires, triggering a fresh HTTP request to the router endpoint on the subsequent lookup.

### What routing policies can I use when selecting a provider?

ML Intern supports three routing policies defined in `_ROUTING_POLICIES`: `fastest`, `cheapest`, and `preferred`. You activate these by appending tags to your model selection (e.g., `model:fastest` or `model:cheapest`). Alternatively, you can specify a concrete provider name as the tag to force usage of a specific backend.

### What happens if the HF router catalog endpoint is unreachable?

If the HTTP request to `https://router.huggingface.co/v1/models` fails, the system checks for a stale cached version of the catalog. If a stale cache exists, it returns that data to maintain functionality. If no cache exists, it returns an empty catalog, allowing the system to continue operating without crashing, though provider lookups will return no results until connectivity is restored.

### Does ML Intern implement its own provider ranking algorithm?

No. ML Intern does not embed its own ranking algorithm for provider selection. Instead, it downloads the up-to-date provider list from the HF router, exposes live providers and their attributes through the CLI, and lets the router's own backend pick the fastest, cheapest, or preferred provider based on the user's tag when the first inference request is sent.