# How to Implement LLM-Based Classifier Routing for Intelligent Request Distribution in Switchyard

> Implement LLM-based classifier routing in Switchyard. Use the llm_task_classifier algorithm for intelligent request distribution and content-aware traffic management across your model infrastructure.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Use Switchyard's `llm_task_classifier` algorithm to let a language model examine incoming requests and return the target route, enabling content-aware traffic distribution across your model infrastructure.**

Switchyard's routing layer can delegate routing decisions to a **language model (LLM) classifier** that intelligently examines request content and selects the optimal backend. This capability is implemented in the `llm_task_classifier` algorithm, available as a Python wrapper around a Rust backend in the NVIDIA-NeMo/Switchyard repository.

## How LLM Classifier Routing Works Internally

The classifier routing system consists of four core components working together to process each request:

| Component | Role | Source Location |
|-----------|------|---------------|
| **Python wrapper** | Exposes `switchyard.libsy.algorithms.llm_task_classifier` which builds a `RoutingDecision` object | [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) |
| **Rust backend** | Executes the LLM call, applies fallbacks, and returns the selected route name | [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) |
| **Deployment TOML** | Declares routing algorithms for each model via the `router` field | `examples/experimental/litellm/` fixtures |
| **CLI launcher** | Reads TOML configuration and constructs the routing pipeline | [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py) |

### Request Processing Flow

When a request arrives at the server:

1. The server builds a `RoutingContext` containing the request payload and configured extra fields.
2. The classifier algorithm receives this context, formats a prompt (or uses a user-provided override), and sends it to the configured LLM endpoint (e.g., Gemini 3.5 Flash, OpenAI GPT-4o).
3. The LLM response is parsed—default format is a single route identifier string, but JSON objects are supported for richer metadata.
4. The selected route passes to the next stage or directly to the backend if this is the final decision.
5. Classification statistics (tokens, cost, latency) are added to the routing-stats JSON for cost estimation and monitoring.

## Setting Up a Classifier Routing Pipeline

Below is a complete, runnable example that demonstrates configuring a classifier model, plugging it into a routing chain, and launching the server.

```python

# tutorial_classifier_routing.py

from pathlib import Path
import asyncio

from switchyard.libsy import algorithms as alg
from switchyard import launch
from switchyard_rust.libsy import LlmFallback

# 1. Configure the LLM that will act as the classifier.

# The default is Gemini "gcp", but any OpenAI-compatible endpoint works

# by setting OPENAI_API_KEY and OPENAI_BASE_URL environment variables.

classifier = LlmFallback(
    model="gemini-3.5-flash",  # model name known to the backend

    max_tokens=256,            # keep classification calls inexpensive

    temperature=0.0,           # deterministic routing decisions

)

# 2. Build the classifier routing algorithm with optional custom prompt.

classifier_algo = alg.llm_task_classifier(
    classifier=classifier,
    # Optional: override the default system prompt

    # prompt="You are a router. Return ONLY the route name.

    # Routes: A, B, C.\n\nRequest: {request}",

    response_format="string",  # "string" (default) or "json"

)

# 3. Create a fallback algorithm for when classification fails.

fallback_algo = alg.random()

# 4. Combine in a stage router: classifier first, then fallback.

router = alg.stage_router(
    stages=[classifier_algo, fallback_algo],
    stop_on_success=True,  # stop when any stage returns a valid route

)

# 5. Launch the server with the custom router.

async def main() -> None:
    await launch(
        model="openrouter",
        router=router,
        config=Path("examples/experimental/litellm/toml/example.toml"),
    )

if __name__ == "__main__":
    asyncio.run(main())

```

### Key Implementation Details

- **`LlmFallback`** from `switchyard_rust.libsy` represents the classification LLM.
- **`alg.llm_task_classifier`** creates a routing algorithm the server can invoke per-request.
- **`alg.stage_router`** chains multiple algorithms, stopping at the first successful route selection.
- The `launch` function automatically wires the router into the server's request handling path.

## Advanced Configuration: Custom Prompts and JSON Output

For richer routing decisions—including confidence scores or multiple metadata fields—configure the classifier to return structured JSON:

```python
classifier_algo = alg.llm_task_classifier(
    classifier=classifier,
    prompt=(
        "You are a routing engine. Given the user request below, output "
        "a JSON object with two fields: 'route' (the chosen route name) "
        "and 'confidence' (a float between 0 and 1). Routes: A, B, C.\n\n"
        "Request: {request}"
    ),
    response_format="json",  # forces JSON parsing

)

```

The server extracts the `route` field automatically and can expose additional fields like `confidence` in routing statistics for downstream observability.

## Monitoring Classifier Usage and Costs

When running with the `--routing-stats-json` CLI flag, Switchyard outputs detailed classification metrics:

```json
{
  "models": { },
  "classifier": {
    "model": "gemini-3.5-flash",
    "request_tokens": 45,
    "response_tokens": 12,
    "cost_estimate": { "total_cost": 0.000018 },
    "total_requests": 1
  }
}

```

Feed this data to `switchyard cost-estimator` to analyze classification overhead, or visualize it in `switchyard live-stats-footer` for real-time monitoring.

## Production Deployment Patterns

### Pattern 1: Direct CLI Launch with Custom Router

```bash
export OPENAI_API_KEY="sk-..."
switchyard launch claude --model my-route \
    --config examples/experimental/litellm/toml/example.toml \
    --router python:/path/to/tutorial_classifier_routing.py

```

### Pattern 2: Stage Router with Confidence-Based Fallback

```python
router = alg.stage_router(
    stages=[
        alg.llm_task_classifier(
            classifier=classifier,
            prompt=(
                "Classify and output JSON with keys "
                "'route' and 'confidence'.\n\nRequest: {request}"
            ),
            response_format="json",
        ),
        alg.random(),
    ],
    stop_on_success=True,
)

```

## Core Source Files Reference

| File | Purpose |
|------|---------|
| [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) | Python routing algorithms including `llm_task_classifier` |
| [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) | Rust implementation of the LLM classification call |
| [`switchyard/cli/launch_command.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launch_command.py) | CLI parsing for `--router` arguments and pipeline construction |
| [`examples/experimental/litellm/src/switchyard_litellm/client.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/experimental/litellm/src/switchyard_litellm/client.py) | Example client for end-to-end classifier testing |
| [`tests/test_libsy_minimal_bindings.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/tests/test_libsy_minimal_bindings.py) | Unit tests for classifier algorithm with various response formats |

## Summary

- **LLM-based classifier routing** in Switchyard uses the `llm_task_classifier` algorithm to make intelligent, content-aware routing decisions.
- The system combines a **Python wrapper** in [`switchyard/libsy/algorithms.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/libsy/algorithms.py) with a **Rust backend** in [`switchyard_rust/libsy.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy.py) for performance.
- Configure the classifier with `LlmFallback` models, custom prompts, and `response_format="json"` for structured outputs.
- Chain classifiers with fallback algorithms using `alg.stage_router` for resilient production deployments.
- Monitor classification costs and performance through built-in routing statistics compatible with Switchyard's cost estimation tools.

## Frequently Asked Questions

### What LLM providers does Switchyard support for classifier routing?

Switchyard supports any OpenAI-compatible endpoint through environment variables (`OPENAI_API_KEY`, `OPENAI_BASE_URL`). The default configuration uses Google's Gemini models via the "gcp" provider. You specify the provider and model name in the `LlmFallback` constructor.

### How does Switchyard handle classifier failures or unknown routes?

The `stage_router` algorithm supports fallback chains. When `stop_on_success=True` is configured, the router proceeds through stages until one returns a valid route. Place deterministic fallbacks like `alg.random()` or `alg.noop()` after the classifier to ensure requests always reach a backend.

### Can I use classifier routing with response caching or other optimizations?

Yes. The classifier routing integrates with Switchyard's broader routing pipeline. You can combine `llm_task_classifier` with stage routers, random selection, and other algorithms in any order. Classification statistics are tracked separately in routing-stats JSON without interfering with other optimizations.

### What is the performance overhead of LLM-based routing?

Each classification call adds one LLM inference request to your critical path. Minimize overhead by using small, fast models (e.g., Gemini 3.5 Flash with `max_tokens=256`), setting `temperature=0.0` for deterministic caching, and monitoring costs through the built-in `classifier` statistics block in routing output.