Composite Routing in NVIDIA Switchyard: Combining LLM Classifier and Stage Router Cascades

Switchyard implements composite routing by stacking an LLM‑based judge (LlmTaskClassifier) over a stage router (StageRouter), using a TierSetter processor to translate the classifier’s verdict into a routing tier that determines the final model pool.

NVIDIA NeMo Switchyard is a Rust-based inference routing framework that enables intelligent request distribution through composite routing patterns. This architecture chains multiple algorithms—specifically an LLM classifier and a tiered stage router—to dynamically route requests to the most appropriate model based on content analysis. By examining the implementation in crates/libsy/src/algorithms/composite.rs, we can trace how these components form a cohesive cascade.

Architecture of the Composite Router

The Composite Router defined in crates/libsy/src/algorithms/composite.rs orchestrates a three-part pipeline: an LLM classifier acting as a judge, a TierSetter processor that bridges decisions to routing state, and a stage router that executes the final selection.

The Judge (LLM Classifier)

The judge is an instance of LlmTaskClassifier implemented in crates/libsy/src/algorithms/llm_class.rs. It evaluates incoming requests and returns a category—either Capable or Efficient—indicating which tier of service is required. The classifier is configured via TaskClassifierConfig and executes only on specific ClassifyTrigger events defined in crates/libsy/src/algorithms/util/affinity.rs: UserTurn (first user message) or NewSession (new routing identity). This prevents the costly LLM evaluation from running on every single request.

The TierSetter Bridge

TierSetter serves as the critical bridge between the judge and the stage router. After the judge returns a category, TierSetter extracts the winning verdict, translates it to a corresponding Tier enum variant (Category::Capable maps to Tier::Capable, Category::Efficient maps to Tier::Efficient), and invokes set_fall_open to configure the stage router’s default tier. The implementation caches the selected tier in a Mutex<HashMap> keyed by routing identity (session ID), ensuring that subsequent requests in the same session reuse the last verdict without re-invoking the classifier.

The Stage Router

The final component is the Stage Router, constructed from StageRouterConfig in crates/libsy/src/algorithms/stage.rs. Under normal operation, this router selects models from tier-specific pools. In the composite pattern, the stage router receives its tier assignment from TierSetter rather than static configuration, effectively using the classifier’s output as its routing imperative.

Execution Flow and Processor Stack

The CompositeRouter struct wires these components together using a fall-through pattern. It implements the Algorithm trait and delegates execution to a FallThrough processor (defined in crates/libsy/src/algorithms/fall_through.rs) that chains the judge, TierSetter, and stage route into a single execution pipeline.

As shown in the source, the construction looks like this:

let setter = TierSetter {
    judge,
    trigger,
    message_hash_fallback,
    tiers: Mutex::new(HashMap::new())
};
let route = build_stage_route(config.stage)?
    .with_name("composite")
    .with_processor(Arc::new(setter));
CompositeRouter { route }

The FallThrough processor ensures sequential execution: the judge classifies, TierSetter updates the tier state, and the stage router selects the final model from the appropriate pool.

Configuring Composite Routing

Switchyard supports both programmatic and declarative configuration for composite routes.

Programmatic Configuration

To build a composite router in Rust, assemble the CompositeRouterConfig with both judge and stage configurations:

use switchyard_libsy::algorithms::composite::{
    CompositeRouter, CompositeRouterConfig,
};
use switchyard_libsy::algorithms::util::affinity::ClassifyTrigger;
use switchyard_libsy::algorithms::stage::StageRouterConfig;
use switchyard_libsy::algorithms::llm_class::{TaskClassifierConfig, LlmClassifierConfig};

let composite_cfg = CompositeRouterConfig {
    // Judge configuration – runs on the first user turn of a session
    judge: TaskClassifierConfig {
        classify_trigger: ClassifyTrigger::UserTurn,
        message_hash_fallback: true,
        ..Default::default()
    },
    // Stage router configuration – maps tiers to model pools
    stage: StageRouterConfig {
        tier_models: vec![
            (Category::Capable, vec!["strong".into()]),
            (Category::Efficient, vec!["weak".into()]),
        ]
        .into_iter()
        .collect(),
        ..Default::default()
    },
};

let router = CompositeRouter::new(composite_cfg).expect("valid composite router");

TOML Configuration

For deployments using switchyard-runner, define the cascade in TOML:

[targets.classifier]
id = "model/classifier"
type = "llm_classifier"
classifier_target = "judge"

[targets.strong]
id = "model/strong"
type = "llm"

[targets.weak]
id = "model/weak"
type = "llm"

[routes.composite]
id = "switchyard/composite"
type = "composite"

[routes.composite.classifier]
target = "classifier"          # the judge

[routes.composite.stage]
target = "stage"               # the stage router

[routes.stage]
type = "stage"
[routes.stage.tier_capable]
models = ["model/strong"]
[routes.stage.tier_efficient]
models = ["model/weak"]

This configuration mirrors the programmatic structure: the composite route references a classifier target (the judge) and a stage target, with the stage router mapping each tier to specific model endpoints.

Summary

• Composite routing in Switchyard chains an LlmTaskClassifier (judge) with a StageRouter via the TierSetter processor, as implemented in crates/libsy/src/algorithms/composite.rs.
• The judge returns content categories (Capable or Efficient) which TierSetter translates into routing tiers and caches per session.
• ClassifyTrigger logic in util/affinity.rs limits LLM invocations to UserTurn or NewSession events, preventing performance overhead on every request.
• The Stage Router consumes the tier set by set_fall_open to select from tier-specific model pools defined in StageRouterConfig.
• Both Rust APIs and TOML configurations support building these cascades, parsed by crates/switchyard-runner/src/config.rs.

Frequently Asked Questions

What is the role of the TierSetter in Switchyard's composite routing?

The TierSetter acts as a stateful bridge between the LLM classifier and the stage router. It receives the judge’s category output, translates it into a Tier enum variant, calls set_fall_open to configure the stage router’s default tier, and caches this decision per routing identity to avoid redundant classification.

How does Switchyard prevent the LLM classifier from running on every request?

The implementation uses ClassifyTrigger configurations (UserTurn or NewSession) defined in crates/libsy/src/algorithms/util/affinity.rs. The composite router explicitly disallows the EveryRequest trigger, ensuring the expensive LlmTaskClassifier only evaluates requests at conversation start or session creation.

Can composite routes be configured without writing Rust code?

Yes. The switchyard-runner binary parses composite routing definitions from TOML configuration files, as shown in crates/switchyard-runner/src/config.rs. You can define the judge, stage router, and tier mappings declaratively in the configuration file.

Which Rust trait must the Composite Router implement to integrate with Switchyard?

The CompositeRouter implements the Algorithm trait, allowing it to function as a first-class routing strategy within the Switchyard framework. It delegates actual execution to a FallThrough processor that manages the internal cascade.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}