How to Debug Routing Failures Using Switchyard-Server Routing Log Files

Enable the --routing-log flag when starting Switchyard-Server to capture per-request JSONL records, then inspect entries for empty model fields or fallback reasons to identify why LLM requests failed to route.

The NVIDIA NeMo Switchyard project provides a durable, line-delimited JSON logging system that records every routing decision made by the LLM gateway. By analyzing these logs, operators can trace failed routing attempts back to specific sessions, identify misconfigured routes, and correlate token usage patterns with routing errors.

Enabling Routing Log Capture

Routing logs are disabled by default. You must explicitly configure the server to write the JSONL file using either the CLI flag or programmatic configuration.

CLI Method

The simplest approach uses the --routing-log option defined in crates/switchyard-server/src/cli.rs:

switchyard-server \
  --config routes.toml \
  --routing-log /var/log/switchyard/routing.jsonl

Programmatic Method

For custom server builds, call ServerState::with_routing_log(path) in crates/switchyard-server/src/lib.rs. This creates a SharedRoutingLog holding an Arc<Mutex<RoutingLog>>:

use switchyard_server::ServerState;
use std::path::Path;

let server = ServerState::new(config)
    .with_routing_log(Path::new("/var/log/switchyard/routing.jsonl"))?;

The constructor ensures the target directory exists and opens the file in append-only mode using fs::File with create(true).append(true) flags, guaranteeing durability even if the process crashes.

Understanding the Routing Log Structure

Each line in the routing log represents a RoutingRecord struct serialized as JSON. The logging pipeline in crates/switchyard-server/src/routing_log.rs follows three stages:

Stage 1: Context Extraction

For every incoming request, the server builds a RoutingLogContext from HTTP headers and the Metadata object. This extracts:

• task — optional workflow identifier
• trial_id — A/B test or experiment identifier
• session_id — primary grouping key (falls back to proxy_x_session_id header for backward compatibility)

The implementation at lines 99–124 handles header parsing with graceful degradation when optional fields are absent.

Stage 2: Record Appending

When a request completes, the server calls SharedRoutingLog::append(context, model, tier, usage). This method:

1. Acquires the mutex lock on the underlying RoutingLog
2. Serializes the RoutingRecord to JSON
3. Writes the line with a trailing newline
4. Releases the lock

Lock acquisition failures are logged but do not abort the request, ensuring the server remains available even under logging pressure.

Stage 3: Snapshot Retrieval

The routing_log::snapshot(path, session_id) function provides read-side analysis. It lazily scans the JSONL file line-by-line at lines 73–95, deserializing each record and aggregating statistics only for the specified session. Malformed lines trigger continue rather than panic, making the reader tolerant of partially-written records during active server operation.

Identifying Routing Failures in Log Entries

Failed routing attempts manifest through specific field patterns in the JSON records:

Failure Pattern	Indicator Field	Interpretation
No available model	"model": "" or missing	Router found no capable backend for the request
Cache miss with fallback	"fallback_reason": "cache_miss"	Request eligible for caching but no hit; routed to alternative
Tier exhaustion	"tier": "fallback" with zero tokens	Primary tier saturated; emergency routing activated
Configuration error	null routing fields	Route table lookup failed or returned invalid entry

Use jq for ad-hoc filtering:

# Find all routing failures for a specific session

jq 'select(.session_id=="abc123" and .model=="")' /var/log/switchyard/routing.jsonl

# Find fallback reasons across all sessions

jq 'select(.fallback_reason!=null) | {session_id, fallback_reason, timestamp}' /var/log/switchyard/routing.jsonl

Extracting Session Snapshots for Deep Analysis

For programmatic debugging, use the snapshot API to aggregate token statistics and call patterns per session:

use switchyard_server::routing_log::{snapshot, SessionStatsSnapshot};
use std::path::Path;

fn diagnose_routing(path: &Path, session_id: &str) {
    match snapshot(path, session_id) {
        Ok(Some(stats)) => {
            println!("Total calls: {}", stats.total_calls);
            println!("Prompt tokens: {}", stats.total_prompt_tokens);
            println!("Completion tokens: {}", stats.total_completion_tokens);
            
            // Per-model breakdown reveals routing distribution
            for (model, count) in &stats.per_model_calls {
                println!("  {}: {} calls", model, count);
            }
        }
        Ok(None) => eprintln!("No records found for session {}", session_id),
        Err(e) => eprintln!("Log read error: {}", e),
    }
}

The SessionStatsSnapshot returned by snapshot includes:

• total_calls — number of requests in session
• total_prompt_tokens / total_completion_tokens — aggregated token usage
• per_model_calls — HashMap<String, usize> showing which models served the session

An empty or unexpectedly small per_model_calls map with high total_calls indicates repeated routing failures where no model could be assigned.

Python Analysis for Operations Teams

For teams preferring Python for log analysis:

import json
from collections import defaultdict
from dataclasses import dataclass
from typing import Optional


@dataclass
class RoutingStats:
    total_calls: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    model_calls: dict = None
    failures: list = None
    
    def __post_init__(self):
        if self.model_calls is None:
            self.model_calls = defaultdict(int)
        if self.failures is None:
            self.failures = []


def analyze_session(log_path: str, session_id: str) -> RoutingStats:
    stats = RoutingStats()
    
    with open(log_path, "r") as f:
        for line_num, line in enumerate(f, 1):
            try:
                rec = json.loads(line)
                if rec.get("session_id") != session_id:
                    continue
                
                stats.total_calls += 1
                stats.prompt_tokens += rec.get("prompt_tokens", 0)
                stats.completion_tokens += rec.get("completion_tokens", 0)
                
                model = rec.get("model", "")
                if model:
                    stats.model_calls[model] += 1
                else:
                    # Capture routing failure with context

                    stats.failures.append({
                        "line": line_num,
                        "timestamp": rec.get("timestamp"),
                        "fallback_reason": rec.get("fallback_reason"),
                        "tier": rec.get("tier")
                    })
                    
            except json.JSONDecodeError:
                continue  # Mirror Rust implementation's graceful handling

    
    return stats


# Usage

stats = analyze_session("/var/log/switchyard/routing.jsonl", "abc123")
print(f"Success rate: {(stats.total_calls - len(stats.failures)) / stats.total_calls:.1%}")
for failure in stats.failures:
    print(f"Routing failure at line {failure['line']}: {failure.get('fallback_reason', 'unknown')}")

Correlating Logs with Runtime Metrics

Routing logs provide decision-level detail, but root-cause analysis often requires combining with the server's Prometheus metrics at /metrics. Key correlations:

• High latency + empty model field — indicates timeout during backend discovery; check switchyard_backend_health metric
• Fallback reason present + cache hit rate drop — verify switchyard_cache_hits_total decreased for the relevant tier
• Token count mismatch — compare usage.prompt_tokens in log with switchyard_tokens_processed_total to detect accounting errors

Summary

• Enable routing logs via --routing-log <PATH> CLI flag or ServerState::with_routing_log() programmatically
• Inspect failure indicators including empty model fields, fallback_reason keys, and tier="fallback" values
• Use snapshot() API to aggregate session-level statistics without parsing the entire log file
• Leverage append-only safety — read and analyze logs while the server runs without synchronization overhead
• Combine with Prometheus — correlate routing decisions with backend health and cache metrics for complete diagnosis

Frequently Asked Questions

What file format does Switchyard-Server use for routing logs?

Switchyard-Server writes line-delimited JSON (JSONL) files. Each line is an independent RoutingRecord struct serialized with serde_json, making the format compatible with standard Unix tools like jq, grep, and line-oriented streaming parsers. The append-only design ensures readers never need file locking.

Can I rotate or delete routing logs while the server is running?

Yes. The RoutingLog implementation in crates/switchyard-server/src/routing_log.rs opens the file once at initialization and writes through the retained fs::File descriptor. External log rotation (via logrotate with copytruncate or similar) will not crash the server, though the new file descriptor will not be automatically picked up without a restart.

Why does my session show zero token counts despite successful requests?

Zero token counts in RoutingRecord entries indicate the request completed routing but usage metrics were unavailable. This typically occurs when: (1) the backend returned a non-200 status without usage headers, (2) the request was rejected before tokenization, or (3) the usage_metrics.rs integration failed to populate the Metadata object. Check adjacent log entries for fallback_reason fields to distinguish these cases.

How do I debug a routing failure when no session_id is provided?

When session_id is absent from headers, RoutingLogContext falls back to the legacy proxy_x_session_id header as implemented at lines 99–124. If neither is present, the field will be null in the JSON record. Filter logs by task or trial_id instead, or configure your load balancer to inject identifiable headers before requests reach Switchyard-Server.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}