OpenHuman Model Routing Strategies: Local vs Remote LLM Provider Selection

OpenHuman uses a policy-driven routing layer that classifies tasks by weight (Lightweight, Medium, Heavy), checks local AI health, and applies routing hints to decide whether to execute LLM calls locally via Ollama or remotely via the OpenHuman backend, with automatic fallback on quality failures.

The tinyhumansai/openhuman repository implements a sophisticated model routing system that sits between your application code and concrete LLM providers. This routing layer intelligently distributes inference workloads across local Ollama-compatible servers and remote OpenHuman backends based on task characteristics, system health, and user-defined constraints.

The Three Model Routing Strategies

OpenHuman employs three complementary model routing strategies to direct tasks to the appropriate LLM provider. Each strategy addresses different deployment scenarios and operational requirements.

Intelligent Routing

Intelligent routing is the default strategy when config.local_ai.runtime_enabled is true and no explicit model routes are configured. This strategy uses the policy::decide function in src/openhuman/routing/policy.rs to dynamically select between local and remote providers.

The system maps model hints (e.g., "hint:reaction", "hint:reasoning") to TaskCategory values—Lightweight, Medium, or Heavy—then evaluates local health status and RoutingHints to determine the primary target and optional fallback. Heavy tasks always route remotely unless privacy constraints force local execution, while lightweight tasks prefer local inference when health checks pass.

Explicit Routing

Explicit routing bypasses the policy engine entirely when administrators define specific model routes in the configuration file. In src/openhuman/providers/ops.rs, the create_routed_provider function constructs a RouterProvider that maps specific hints directly to remote model names.

For example, a configuration entry might route "hint:agentic" directly to gpt-4o-mini, ensuring consistent provider selection regardless of task weight or local system status. This strategy is ideal when you require deterministic model selection for specific use cases.

Remote-Only Fallback

Remote-only fallback activates when config.local_ai.runtime_enabled is false or when the health probe in src/openhuman/routing/health.rs determines that the local Ollama server is unreachable. In this mode, all requests route to the remote OpenHuman backend through the OpenAiCompatibleProvider, ensuring service continuity even when local infrastructure fails.

The Routing Decision Pipeline

The routing layer processes every LLM request through a specific decision flow implemented across multiple source files.

Task Classification

The pipeline begins in src/openhuman/routing/policy.rs, where the classify function converts model hints into task categories:

use openhuman::routing::policy::{classify, TaskCategory};

let category = classify("hint:reaction");
// Returns TaskCategory::Lightweight

This classification determines the baseline routing behavior before other factors intervene.

Health Verification

Before selecting a local provider, the system executes a health check via src/openhuman/routing/health.rs. This probe verifies that the local Ollama instance or OpenAI-compatible server is responsive and capable of accepting requests.

Policy Evaluation

The decide function in src/openhuman/routing/policy.rs evaluates RoutingHints—including privacy_required, latency_budget, and cost_sensitivity—to refine the routing target:

use openhuman::routing::policy::{
    decide, RoutingHints, RoutingTarget, TaskCategory, LatencyBudget, CostSensitivity,
};

let hints = RoutingHints {
    privacy_required: false,
    latency_budget: LatencyBudget::Low,
    cost_sensitivity: CostSensitivity::Normal,
};

let (primary, fallback) = decide(
    TaskCategory::Lightweight,
    "local-llama:7b",
    "remote-reasoning-v1",
    true,  // local health OK
    &hints,
);

Execution with Quality Fallback

The IntelligentRoutingProvider in src/openhuman/routing/provider.rs executes the primary request. If the local provider returns an error or the quality::is_low_quality heuristic judges the response insufficient, the system automatically retries against the fallback target—unless privacy_required is true, which suppresses remote calls entirely.

Implementation Architecture

The routing system relies on several key components working together across the codebase.

Provider Construction

In src/openhuman/providers/ops.rs, the create_intelligent_routing_provider function assembles the provider chain:

let provider = openhuman::providers::ops::create_intelligent_routing_provider(
    Some("https://api.openhuman.com"),
    Some("my-api-key"),
    &config,
    &ProviderRuntimeOptions::default(),
)?;

This function instantiates both a remote OpenAiCompatibleProvider for the OpenHuman backend and a local OpenAiCompatibleProvider for Ollama, wrapping them in an IntelligentRoutingProvider that handles the routing logic.

Explicit Route Configuration

To use explicit routing, define model_routes in your configuration file and call create_routed_provider:

# config.toml

[[model_routes]]
hint = "hint:agentic"
model = "gpt-4o-mini"

let routed = openhuman::providers::ops::create_routed_provider(
    None,
    None,
    &reliability_cfg,
    &config.model_routes,
    "default-remote-model",
)?;

Telemetry and Observability

Every routing decision emits a RoutingRecord via src/openhuman/routing/telemetry.rs, capturing the model hint, task category, selected provider, latency, fallback flags, and token usage. These records appear under the "routing" tracing target, enabling detailed audit trails of provider selection.

Key Source Files

Understanding the complete routing implementation requires familiarity with these specific files:

• src/openhuman/routing/provider.rs - Contains IntelligentRoutingProvider, which implements the Provider trait and handles fallback logic
• src/openhuman/routing/policy.rs - Defines TaskCategory, RoutingHints, and the decide function for policy evaluation
• src/openhuman/routing/factory.rs - Constructs routing providers from configuration objects
• src/openhuman/providers/ops.rs - Public API for creating intelligent and routed providers
• src/openhuman/routing/telemetry.rs - Emits routing decisions for observability
• src/openhuman/routing/health.rs - Performs health probes on local AI runtimes
• src/openhuman/routing/quality.rs - Implements heuristics for detecting low-quality local responses

Summary

• OpenHuman employs policy-driven routing that weighs task characteristics against system health and user constraints to select between local and remote LLM providers.
• Three strategies cover all scenarios: intelligent routing for dynamic selection, explicit routing for deterministic model mapping, and remote-only fallback for high-availability requirements.
• Automatic quality fallback allows the system to retry failed or low-quality local inferences against remote backends unless privacy requirements prohibit external calls.
• Comprehensive telemetry via RoutingRecord events enables full observability of routing decisions under the "routing" tracing target.
• Configuration-driven architecture lets administrators define specific model routes in TOML while maintaining intelligent defaults for unspecified hints.

Frequently Asked Questions

How does OpenHuman decide between local and remote LLM providers?

OpenHuman uses the decide function in src/openhuman/routing/policy.rs to evaluate task categories (Lightweight, Medium, Heavy), local health status, and routing hints. Heavy tasks route remotely by default, lightweight tasks prefer local execution when healthy, and medium tasks consider latency budgets and cost sensitivity to determine the optimal provider.

What happens if the local Ollama server fails during a request?

If the primary target is local and the call errors or produces low-quality output (as determined by quality::is_low_quality in src/openhuman/routing/quality.rs), the IntelligentRoutingProvider automatically retries the request against the remote fallback target. This transparent fallback ensures request completion unless privacy_required is set to true in the routing hints.

Can I force specific model hints to use specific providers?

Yes, by defining model_routes in your configuration file and using create_routed_provider from src/openhuman/providers/ops.rs, you can map specific hints directly to concrete models. This explicit routing bypasses the policy engine entirely, ensuring deterministic provider selection for critical workflows.

What routing information is available for monitoring?

The routing::telemetry module emits RoutingRecord events for every routing decision, including the model hint, TaskCategory, selected provider, latency metrics, fallback flags, and token usage. These records are emitted under the "routing" tracing target, compatible with standard Rust logging infrastructure for audit and monitoring purposes.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}