How the HF Router Catalog Auto-Selects Providers for Different Models in ML Intern

ML Intern uses the Hugging Face Inference Router to fetch a live catalog of available providers every five minutes, then applies user-specified routing policies (fastest, cheapest, preferred) to filter live providers while letting the router backend make the final selection.

The huggingface/ml-intern repository implements an intelligent routing layer that automatically discovers which cloud providers can serve a given model. By leveraging the HF router catalog, the system eliminates manual provider configuration and ensures optimal model serving through real-time metadata caching and policy-based selection.

Fetching and Caching the Router Catalog

The auto-selection process begins in agent/core/hf_router_catalog.py, which communicates with the Hugging Face Inference Router's public endpoint at https://router.huggingface.co/v1/models. This endpoint returns a JSON list containing every available model along with provider-level metadata, including status, context length, pricing, and tool-call support.

The _fetch_catalog() function retrieves this data with a short HTTP timeout and stores it in a module-level cache (_cache) for 300 seconds (_CACHE_TTL_SECONDS = 300). This five-minute TTL guarantees fast, low-latency routing decisions for subsequent lookups. If the HTTP request fails, the system falls back to a stale cache when available; otherwise, it returns an empty catalog to prevent system crashes.

Parsing Provider Metadata into Structured Objects

Once fetched, the raw catalog data undergoes transformation through the _parse_entry() function. This converts each JSON entry into a structured ModelInfo dataclass that contains a list of ProviderInfo objects.

Each ProviderInfo captures critical attributes:

• provider: The provider identifier
• status: Availability state (e.g., live or down)
• context_length: Maximum token context
• input_price and output_price: Cost per million tokens
• supports_tools and supports_structured_output: Capability flags

The ModelInfo class exposes helper properties such as live_providers, max_context_length, and any_supports_tools, enabling downstream components to quickly reason about model availability and capabilities without parsing raw JSON.

Auto-Selection Logic in the Model Switcher

The user-facing routing logic resides in agent/core/model_switcher.py. When a user executes the /model command, the system invokes _print_hf_routing_info() to process the model identifier and optional routing tags.

The function accepts model strings in the format org/model:tag, where the tag can be :fastest, :cheapest, :preferred, or a specific provider name. These correspond to the three supported routing policies defined in _ROUTING_POLICIES = {"fastest", "cheapest", "preferred"}. When a policy tag is provided, the system validates that matching providers exist in live_providers and displays the selected policy.

When no tag is supplied, the default policy is "auto (fastest)", which lists all available live providers. The actual provider selection occurs later when the Hugging Face router service processes the first inference request. The system also emits warnings when no live providers exist for a model or when none advertise tool-call support, giving users the opportunity to intervene before the probe call fails.

Practical Implementation Examples

# Example 1 – Retrieve a model’s provider information

from agent.core import hf_router_catalog as cat

model = "MiniMaxAI/MiniMax-M2.7"
info = cat.lookup(model)          # → ModelInfo or None

if info:
    for p in info.live_providers:
        print(f"{p.provider}: ${p.input_price}/{p.output_price} per M tok "
              f"{p.context_length or 'n/a'} ctx "
              f"{'tools' if p.supports_tools else 'no tools'}")
else:
    # Not in the catalog – suggest close matches

    print("Did you mean:", cat.fuzzy_suggest(model))

# Example 2 – Show routing details as the REPL does

from agent.core import model_switcher as ms
from rich.console import Console

console = Console()
ms._print_hf_routing_info("MiniMaxAI/MiniMax-M2.7:fastest", console)

# Output includes:

#   routing: fastest

#   huggingface: $0.02/$0.06 per M tok, 4,096 ctx, tools

# Example 3 – Pre‑warm the catalog at startup (called in agent/main.py)

import asyncio
from agent.core import hf_router_catalog

asyncio.create_task(asyncio.to_thread(hf_router_catalog.prewarm))

# The catalog is fetched once so the first `/model` lookup is instantaneous.

Summary

• The HF router catalog is fetched from https://router.huggingface.co/v1/models every five minutes and cached in memory via _cache to ensure low-latency lookups.
• Provider metadata is structured into ModelInfo and ProviderInfo dataclasses exposing properties like live_providers and max_context_length.
• Users can specify routing policies (fastest, cheapest, preferred) via tags appended to model names, or rely on the default "auto (fastest)" behavior.
• ML Intern delegates the final provider selection to the Hugging Face router backend, which chooses the actual provider when processing inference requests.
• The system gracefully handles failures by falling back to stale cache data and warns users about unavailable providers or missing capability support.

Frequently Asked Questions

How often does ML Intern refresh the HF router catalog?

The catalog refreshes every five minutes. The _fetch_catalog() function in agent/core/hf_router_catalog.py implements a time-based cache with _CACHE_TTL_SECONDS = 300 that automatically expires, triggering a fresh HTTP request to the router endpoint on the subsequent lookup.

What routing policies can I use when selecting a provider?

ML Intern supports three routing policies defined in _ROUTING_POLICIES: fastest, cheapest, and preferred. You activate these by appending tags to your model selection (e.g., model:fastest or model:cheapest). Alternatively, you can specify a concrete provider name as the tag to force usage of a specific backend.

What happens if the HF router catalog endpoint is unreachable?

If the HTTP request to https://router.huggingface.co/v1/models fails, the system checks for a stale cached version of the catalog. If a stale cache exists, it returns that data to maintain functionality. If no cache exists, it returns an empty catalog, allowing the system to continue operating without crashing, though provider lookups will return no results until connectivity is restored.

Does ML Intern implement its own provider ranking algorithm?

No. ML Intern does not embed its own ranking algorithm for provider selection. Instead, it downloads the up-to-date provider list from the HF router, exposes live providers and their attributes through the CLI, and lets the router's own backend pick the fastest, cheapest, or preferred provider based on the user's tag when the first inference request is sent.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}