How to Implement LLM-Based Classifier Routing for Intelligent Request Distribution in Switchyard

Use Switchyard's llm_task_classifier algorithm to let a language model examine incoming requests and return the target route, enabling content-aware traffic distribution across your model infrastructure.

Switchyard's routing layer can delegate routing decisions to a language model (LLM) classifier that intelligently examines request content and selects the optimal backend. This capability is implemented in the llm_task_classifier algorithm, available as a Python wrapper around a Rust backend in the NVIDIA-NeMo/Switchyard repository.

How LLM Classifier Routing Works Internally

The classifier routing system consists of four core components working together to process each request:

Component	Role	Source Location
Python wrapper	Exposes switchyard.libsy.algorithms.llm_task_classifier which builds a RoutingDecision object	switchyard/libsy/algorithms.py
Rust backend	Executes the LLM call, applies fallbacks, and returns the selected route name	switchyard_rust/libsy.py
Deployment TOML	Declares routing algorithms for each model via the router field	examples/experimental/litellm/ fixtures
CLI launcher	Reads TOML configuration and constructs the routing pipeline	switchyard/cli/launch_command.py

Request Processing Flow

When a request arrives at the server:

1. The server builds a RoutingContext containing the request payload and configured extra fields.
2. The classifier algorithm receives this context, formats a prompt (or uses a user-provided override), and sends it to the configured LLM endpoint (e.g., Gemini 3.5 Flash, OpenAI GPT-4o).
3. The LLM response is parsed—default format is a single route identifier string, but JSON objects are supported for richer metadata.
4. The selected route passes to the next stage or directly to the backend if this is the final decision.
5. Classification statistics (tokens, cost, latency) are added to the routing-stats JSON for cost estimation and monitoring.

Setting Up a Classifier Routing Pipeline

Below is a complete, runnable example that demonstrates configuring a classifier model, plugging it into a routing chain, and launching the server.

# tutorial_classifier_routing.py

from pathlib import Path
import asyncio

from switchyard.libsy import algorithms as alg
from switchyard import launch
from switchyard_rust.libsy import LlmFallback

# 1. Configure the LLM that will act as the classifier.

# The default is Gemini "gcp", but any OpenAI-compatible endpoint works

# by setting OPENAI_API_KEY and OPENAI_BASE_URL environment variables.

classifier = LlmFallback(
    model="gemini-3.5-flash",  # model name known to the backend

    max_tokens=256,            # keep classification calls inexpensive

    temperature=0.0,           # deterministic routing decisions

)

# 2. Build the classifier routing algorithm with optional custom prompt.

classifier_algo = alg.llm_task_classifier(
    classifier=classifier,
    # Optional: override the default system prompt

    # prompt="You are a router. Return ONLY the route name.

    # Routes: A, B, C.\n\nRequest: {request}",

    response_format="string",  # "string" (default) or "json"

)

# 3. Create a fallback algorithm for when classification fails.

fallback_algo = alg.random()

# 4. Combine in a stage router: classifier first, then fallback.

router = alg.stage_router(
    stages=[classifier_algo, fallback_algo],
    stop_on_success=True,  # stop when any stage returns a valid route

)

# 5. Launch the server with the custom router.

async def main() -> None:
    await launch(
        model="openrouter",
        router=router,
        config=Path("examples/experimental/litellm/toml/example.toml"),
    )

if __name__ == "__main__":
    asyncio.run(main())

Key Implementation Details

• LlmFallback from switchyard_rust.libsy represents the classification LLM.
• alg.llm_task_classifier creates a routing algorithm the server can invoke per-request.
• alg.stage_router chains multiple algorithms, stopping at the first successful route selection.
• The launch function automatically wires the router into the server's request handling path.

Advanced Configuration: Custom Prompts and JSON Output

For richer routing decisions—including confidence scores or multiple metadata fields—configure the classifier to return structured JSON:

classifier_algo = alg.llm_task_classifier(
    classifier=classifier,
    prompt=(
        "You are a routing engine. Given the user request below, output "
        "a JSON object with two fields: 'route' (the chosen route name) "
        "and 'confidence' (a float between 0 and 1). Routes: A, B, C.\n\n"
        "Request: {request}"
    ),
    response_format="json",  # forces JSON parsing

)

The server extracts the route field automatically and can expose additional fields like confidence in routing statistics for downstream observability.

Monitoring Classifier Usage and Costs

When running with the --routing-stats-json CLI flag, Switchyard outputs detailed classification metrics:

{
  "models": { },
  "classifier": {
    "model": "gemini-3.5-flash",
    "request_tokens": 45,
    "response_tokens": 12,
    "cost_estimate": { "total_cost": 0.000018 },
    "total_requests": 1
  }
}

Feed this data to switchyard cost-estimator to analyze classification overhead, or visualize it in switchyard live-stats-footer for real-time monitoring.

Production Deployment Patterns

Pattern 1: Direct CLI Launch with Custom Router

export OPENAI_API_KEY="sk-..."
switchyard launch claude --model my-route \
    --config examples/experimental/litellm/toml/example.toml \
    --router python:/path/to/tutorial_classifier_routing.py

Pattern 2: Stage Router with Confidence-Based Fallback

router = alg.stage_router(
    stages=[
        alg.llm_task_classifier(
            classifier=classifier,
            prompt=(
                "Classify and output JSON with keys "
                "'route' and 'confidence'.\n\nRequest: {request}"
            ),
            response_format="json",
        ),
        alg.random(),
    ],
    stop_on_success=True,
)

Core Source Files Reference

File	Purpose
switchyard/libsy/algorithms.py	Python routing algorithms including llm_task_classifier
switchyard_rust/libsy.py	Rust implementation of the LLM classification call
switchyard/cli/launch_command.py	CLI parsing for --router arguments and pipeline construction
examples/experimental/litellm/src/switchyard_litellm/client.py	Example client for end-to-end classifier testing
tests/test_libsy_minimal_bindings.py	Unit tests for classifier algorithm with various response formats

Summary

• LLM-based classifier routing in Switchyard uses the llm_task_classifier algorithm to make intelligent, content-aware routing decisions.
• The system combines a Python wrapper in switchyard/libsy/algorithms.py with a Rust backend in switchyard_rust/libsy.py for performance.
• Configure the classifier with LlmFallback models, custom prompts, and response_format="json" for structured outputs.
• Chain classifiers with fallback algorithms using alg.stage_router for resilient production deployments.
• Monitor classification costs and performance through built-in routing statistics compatible with Switchyard's cost estimation tools.

Frequently Asked Questions

What LLM providers does Switchyard support for classifier routing?

Switchyard supports any OpenAI-compatible endpoint through environment variables (OPENAI_API_KEY, OPENAI_BASE_URL). The default configuration uses Google's Gemini models via the "gcp" provider. You specify the provider and model name in the LlmFallback constructor.

How does Switchyard handle classifier failures or unknown routes?

The stage_router algorithm supports fallback chains. When stop_on_success=True is configured, the router proceeds through stages until one returns a valid route. Place deterministic fallbacks like alg.random() or alg.noop() after the classifier to ensure requests always reach a backend.

Can I use classifier routing with response caching or other optimizations?

Yes. The classifier routing integrates with Switchyard's broader routing pipeline. You can combine llm_task_classifier with stage routers, random selection, and other algorithms in any order. Classification statistics are tracked separately in routing-stats JSON without interfering with other optimizations.

What is the performance overhead of LLM-based routing?

Each classification call adds one LLM inference request to your critical path. Minimize overhead by using small, fast models (e.g., Gemini 3.5 Flash with max_tokens=256), setting temperature=0.0 for deterministic caching, and monitoring costs through the built-in classifier statistics block in routing output.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}