How to Implement Conversation Affinity and Turn-Based Routing in Switchyard

To implement conversation affinity and turn-based routing in Switchyard, use AffinityRouter for sticky session routing and StageRouter for per-turn decision-making based on tool signals or LLM judges.

Switchyard provides two orthogonal routing mechanisms that work together to optimize multi-turn LLM conversations. Conversation affinity ensures a user's session stays bound to a previously selected model, while turn-based routing dynamically chooses models for each turn based on tool-call outcomes or capability judgments. This guide walks through the implementation using the NVIDIA-NeMo/Switchyard source code.

What Conversation Affinity Does

The AffinityRouter in crates/libsy/src/algorithms/util/affinity.rs implements sticky routing by serving dual roles: as a processor that records the first routing decision, and as a classifier that returns that same decision on subsequent requests.

When enabled, the router:

• Derives a routing identity from session_id + agent_id, or from a hash of the first user message when message_hash_fallback=True
• Stores the selected model on the first Decision event
• Returns a Score with confidence 1.0 for the retained model on subsequent matching requests

If no usable identity exists, the router emits a one-time warning and abstains, letting the outer cascade decide normally.

Enabling Session Affinity in LlmTaskClassifier

The llm_task_classifier Python binding automatically configures AffinityRouter when session_affinity=True in TaskClassifierConfig. In crates/libsy/src/algorithms/llm_class.rs lines 608-618, the construction logic registers the router twice:

from switchyard.libsy import llm_task_classifier, TaskClassifierConfig

classifier = llm_task_classifier(
    judge_target="judge",
    efficient_target="efficient",
    capable_target="capable",
    config=TaskClassifierConfig(
        base_threshold=0.5,
        session_affinity=True,          # Activates AffinityRouter

        message_hash_fallback=True,     # Hash first message if no session

    ),
)

The AffinityRouter::new() call in Rust creates the dual-registered processor/classifier pair via route = route.with_processor(affinity.clone()).with_classifier(affinity). The identity lookup and 1.0 confidence scoring are implemented in affinity.rs lines 163-175 and 188-198 respectively.

How Turn-Based Routing Works

The StageRouter in crates/libsy/src/algorithms/stage.rs builds a FallThrough cascade that examines each turn independently. It decides whether to escalate to a capable tier, stay efficient, or invoke an optional capability judge.

The cascade order in StageRouter::new (lines 29-33) is:

1. ToolSignalProcessor — analyzes recent tool-call results (recent_window parameter)
2. StageClassifier — applies confidence_threshold to signal strength
3. LlmFallback (optional) — capability judge when signals are ambiguous
4. DefaultTarget — picker default tier as final fallback

Construct a stage router via the stage_router Python binding:

from switchyard.libsy import stage_router, LlmFallback, TaskClassifierConfig

# Optional judge runs only when signals are undecided

judge = LlmFallback(
    judge_target="judge",
    config=TaskClassifierConfig(base_threshold=0.5)
)

router = stage_router(
    capable_target="strong",        # Capable tier (e.g., Codex-class model)

    efficient_target="weak",        # Efficient tier (e.g., smaller model)

    picker="efficient_first",       # Default tier when signals weak

    confidence_threshold=0.6,       # Signal confidence for capable tier

    recent_window=3,                # Tool results examined per turn

    escalation_note="Please investigate the failure.",  # Hand-off note

    classifier=judge,               # Optional capability judge

)

The ToolSignalProcessor writes a Decision with a Signal source when confidence exceeds the threshold (lines 172-179 in stage.rs). When this happens, the judge is never invoked—verified in the test "a decisive signal never reaches the judge" (lines 525-538).

If signals are ambiguous, the LlmFallback wrapped in a SourceStamp runs the capability judge (lines 180-192). The final DefaultTarget guarantees a routing decision (lines 194-198).

Combining Both Mechanisms in a Pipeline

For production deployments, chain AffinityRouter outside StageRouter so session stickiness short-circuits before per-turn analysis:

from switchyard.libsy import stage_router, llm_task_classifier, TaskClassifierConfig, LlmFallback

# Capability judge for ambiguous signals

judge = LlmFallback(
    judge_target="judge",
    config=TaskClassifierConfig(base_threshold=0.5, session_affinity=True)
)

# Per-turn routing decisions

stage = stage_router(
    capable_target="strong",
    efficient_target="weak",
    picker="efficient_first",
    confidence_threshold=0.7,
    recent_window=2,
    classifier=judge,
)

# Session affinity wrapper

affinity_classifier = llm_task_classifier(
    judge_target="judge",
    efficient_target="weak",
    capable_target="strong",
    config=TaskClassifierConfig(
        base_threshold=0.5,
        session_affinity=True,
        message_hash_fallback=True,
    ),
)

Runtime flow:

• Affinity check first: If the routing identity matches a stored assignment, return that model immediately with confidence 1.0
• Proceed to stage router: No affinity match means first turn or cleared session
• Tool signal analysis: ToolSignalProcessor examines the recent tool-call window
• Threshold decision: Signals above confidence_threshold → capable tier; decisive routing complete
• Judge fallback: Ambiguous signals trigger the LlmFallback capability judge
• Default picker: If all else abstains, use picker default tier

Sub-Agent Affinity for Flexible Root Traffic

For agent architectures where root traffic should remain flexible but sub-agent conversations need stickiness, use AffinityRouter.for_subagents():

from switchyard.libsy import AffinityRouter

# Only retains assignments for child agents, not root sessions

sub_affinity = AffinityRouter.for_subagents()

# Register in custom FallThrough chain:

# route = FallThrough([...]).with_processor(sub_affinity).with_classifier(sub_affinity)

This keeps the main conversation free to reroute while ensuring delegated sub-tasks maintain model consistency.

Customizing Turn-Based Behavior

Adjust TaskClassifierConfig parameters to control when the capability judge activates:

from switchyard.libsy import LlmFallback, TaskClassifierConfig

custom_judge = LlmFallback(
    judge_target="custom_judge",
    config=TaskClassifierConfig(
        base_threshold=0.4,
        recent_turn_window=4,      # Judge sees last 4 turns for context

    ),
)

Lower base_threshold makes the judge more likely to render a verdict; higher recent_turn_window gives the judge more conversation history but increases token cost and latency.

Key Source Files for Implementation

File	Purpose
crates/libsy/src/algorithms/util/affinity.rs	AffinityRouter with dual processor/classifier implementation
crates/libsy/src/algorithms/llm_class.rs	LlmTaskClassifier constructor; affinity integration
crates/libsy/src/algorithms/stage.rs	StageRouter with ToolSignalProcessor, StageClassifier, LlmFallback
switchyard_rust/libsy.py	Python bindings for stage_router, llm_task_classifier
switchyard/libsy/__init__.py	Public API exports

Summary

• Conversation affinity uses AffinityRouter to bind sessions to models after first selection, eliminating classifier overhead on subsequent turns
• Turn-based routing uses StageRouter to examine tool signals per turn, escalating to capable tiers when confidence thresholds are met
• Enable affinity with session_affinity=True in TaskClassifierConfig passed to llm_task_classifier
• Configure turn routing with stage_router parameters: confidence_threshold, recent_window, optional LlmFallback judge, and picker default
• Combine both by placing the affinity classifier outermost in the algorithm chain for short-circuit efficiency
• Sub-agent variants allow flexible root routing with sticky delegation via AffinityRouter.for_subagents()

Frequently Asked Questions

How does Switchyard handle the first request when no affinity record exists?

The AffinityRouter abstains and emits a one-time warning, allowing the downstream cascade (typically StageRouter) to make a normal routing decision. Once that decision is made, the processor component records it for future identity matches.

What happens when tool signals are decisive versus ambiguous?

When ToolSignalProcessor confidence exceeds confidence_threshold, the StageClassifier immediately selects the capable tier and the optional LlmFallback judge never runs. When signals are below threshold, the cascade falls through to the judge if configured, then to the picker default.

Can I use conversation affinity without turn-based routing?

Yes. The AffinityRouter is independently usable. Construct it via llm_task_classifier with session_affinity=True and any valid targets—the judge_target is ignored when affinity resolves successfully, so it can be a dummy value.

How does message_hash_fallback work when no session ID is present?

Switchyard hashes the first user message content to generate a synthetic identity. This provides sticky routing for stateless clients or purely hash-based conversation grouping, implemented in affinity.rs lines 163-175.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}