How the LLM Classifier in NVIDIA Switchyard Differentiates Between Strong and Weak Tiers

The LLM classifier in Switchyard differentiates between "strong" (capable) and "weak" (efficient) tiers by comparing a judge model's solve probability (p_solve) against a configurable threshold, routing to the weak tier when the probability meets or exceeds that threshold and to the strong tier otherwise.

Switchyard is NVIDIA's open-source routing framework for LLM inference. At its core sits the LlmTaskClassifier algorithm, which makes intelligent routing decisions to optimize the cost-performance trade-off. This article examines how the classifier determines which tier—weak (efficient) or strong (capable)—should handle each incoming request based on the actual implementation in the Switchyard source code.

Classifier Operating Modes

The LlmTaskClassifier supports two distinct modes for tier selection, each implementing the routing_tier method to label requests.

Capability Mode

In capability mode, a dedicated judge model evaluates the task upfront before any inference occurs. The judge returns a TaskClassifierVerdict containing:

• p_solve: The estimated probability that the efficient model can successfully complete the task
• A capability boundary (supported, uncertain, unsupported, or unmatched)
• Rule identifiers for traceability

Escalation Mode

In escalation mode, the system attempts inference with the efficient model first. A judge then evaluates whether that response is sufficient or if the request must "escalate" to the capable model. This mode incurs the latency of potentially two model calls but avoids judge overhead for simple requests that the efficient model handles well.

How the Routing Tier Is Determined

Both classifier variants implement routing_tier to return static string labels based on which model ID was selected. The implementation in crates/libsy/src/algorithms/llm_class.rs reveals the mapping logic.

For capability mode (TaskClassifier):

fn routing_tier(&self, selected_model_id: &ModelId) -> Option<&'static str> {
    if self.efficient_target == self.capable_target {
        None
    } else if *selected_model_id == self.efficient_target {
        Some("weak")
    } else if *selected_model_id == self.capable_target {
        Some("strong")
    } else {
        None
    }
}

For escalation mode (EscalationClassifier):

fn routing_tier(&self, selected_model_id: &ModelId) -> Option<&'static str> {
    if self.capable == self.efficient {
        None
    } else if *selected_model_id == self.capable {
        Some("strong")
    } else if *selected_model_id == self.efficient {
        Some("weak")
    } else {
        None
    }
}

Note that when both targets reference the same model, no tier distinction exists and None is returned.

The Threshold-Based Decision Logic

The core differentiation mechanism lives in TaskClassifierPolicy::to_classification. The policy computes a dynamic threshold from the verdict's capability boundary, then selects the target by comparing p_solve against that threshold.

Threshold Computation

fn threshold(&self, verdict: &TaskClassifierVerdict) -> Option<f64> {
    Some(self.base_threshold + f64::from(verdict.boundary_steps()?) * self.threshold_step)
}

The threshold increases for less certain boundaries:

• Supported tasks use base_threshold directly
• Uncertain tasks add one threshold_step
• Unsupported tasks add two threshold_step values

Target Selection

let target = if verdict.p_solve >= threshold
    || (threshold - verdict.p_solve).abs() <= f64::EPSILON {
    &self.efficient_target
} else {
    &self.capable_target
};

This logic appears in crates/libsy/src/algorithms/llm_class.rs at lines 1010-1016. The epsilon comparison ensures floating-point stability at equality boundaries.

Configuration Parameters

The classifier's behavior is controlled through TaskClassifierConfig, defined at lines 26-34 of llm_class.rs:

pub struct TaskClassifierConfig {
    pub base_threshold: f64,
    pub threshold_step: f64,
    ...
}

Parameter	Purpose	Typical Effect
base_threshold	Minimum solve probability for weak-tier routing of supported tasks	Lower values increase weak-tier usage
threshold_step	Penalty added per uncertainty level	Higher values more aggressively promote uncertain tasks to strong tier

Lowering base_threshold from 0.5 to 0.3, for example, increases traffic to the efficient model but risks more failed completions.

Practical Implementation Examples

Python: Configuring a Capability-Based Classifier

from switchyard.libsy import algorithms as alg
from switchyard.libsy import LlmClassifierConfig, TaskClassifierConfig

cfg = TaskClassifierConfig(
    base_threshold=0.5,      # 50% solve prob needed for weak tier

    threshold_step=0.1,      # extra slack for uncertain/unsupported tasks

    session_affinity=False,
)

classifier = alg.llm_task_classifier(
    LlmClassifierConfig.Capability(
        judge_target="judge",
        efficient_target="efficient",   # weak tier

        capable_target="capable",       # strong tier

        config=cfg,
    )
)

The factory function resides in switchyard/libsy/algorithms.py.

Rust: Direct Classifier Instantiation

let classifier = LlmTaskClassifier::new(
    LlmClassifierConfig::Capability {
        judge_target: ModelId::from("judge"),
        efficient_target: ModelId::from("efficient"),
        capable_target: ModelId::from("capable"),
        config: TaskClassifierConfig {
            base_threshold: 0.6,
            threshold_step: 0.0,
            ..Default::default()
        },
    },
)?;

Construction logic appears in llm_class.rs at lines 650-660.

CLI Integration

from switchyard import launch_command

launch_command.run([
    "claude",
    "--model", "my-route",
    "--config", "routes.toml",
])

The CLI entry point is switchyard/cli/switchyard_cli.py.

Key Implementation Files

File	Role
crates/libsy/src/algorithms/llm_class.rs	Core LlmTaskClassifier, TaskClassifierPolicy, and routing_tier implementations
switchyard/libsy/algorithms.py	Python façade for classifier construction
switchyard/cli/switchyard_cli.py	CLI entry point for pipeline execution

Summary

• Two modes: Capability mode uses upfront judge evaluation; escalation mode tries efficient inference first with fallback
• Threshold mechanism: base_threshold plus threshold_step per uncertainty level determines the cutoff for weak-tier routing
• Decision rule: p_solve >= threshold selects weak (efficient); otherwise strong (capable)
• Explicit mapping: routing_tier methods convert selected ModelId to "weak" or "strong" labels for observability
• Configuration: TaskClassifierConfig exposes tunable parameters without code changes

Frequently Asked Questions

How does the classifier handle cases where the efficient and capable models are identical?

When efficient_target == capable_target, both routing_tier implementations return None (lines 888 and 887 in llm_class.rs). No tier distinction exists since both routing options reference the same underlying model.

What happens when p_solve exactly equals the computed threshold?

The implementation uses (threshold - verdict.p_solve).abs() <= f64::EPSILON to handle floating-point equality, routing to the efficient target in this case. This ensures deterministic behavior at boundary conditions.

Can the same classifier instance switch between capability and escalation modes?

No. The mode is determined at construction time through the LlmClassifierConfig enum variant (Capability or Escalation). Each variant carries distinct fields and instantiates different internal structs (TaskClassifier vs. EscalationClassifier).

Where does the judge model itself run?

The judge is treated as another model target referenced by ModelId. In the examples, "judge" resolves through Switchyard's standard model routing configuration, which may point to a lightweight classifier model or even a rules-based evaluator depending on deployment setup.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}