# How to Debug Routing Failures Using Switchyard-Server Routing Log Files

> Debug routing failures in Switchyard-Server by enabling the routing log flag. Inspect log entries to pinpoint why LLM requests fail to route, identifying empty model fields or fallback reasons.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: how-to-guide
- Published: 2026-08-15

---

**Enable the `--routing-log` flag when starting Switchyard-Server to capture per-request JSONL records, then inspect entries for empty model fields or fallback reasons to identify why LLM requests failed to route.**

The NVIDIA NeMo Switchyard project provides a durable, line-delimited JSON logging system that records every routing decision made by the LLM gateway. By analyzing these logs, operators can trace failed routing attempts back to specific sessions, identify misconfigured routes, and correlate token usage patterns with routing errors.

## Enabling Routing Log Capture

Routing logs are disabled by default. You must explicitly configure the server to write the JSONL file using either the CLI flag or programmatic configuration.

### CLI Method

The simplest approach uses the `--routing-log` option defined in [`crates/switchyard-server/src/cli.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/cli.rs):

```bash
switchyard-server \
  --config routes.toml \
  --routing-log /var/log/switchyard/routing.jsonl

```

### Programmatic Method

For custom server builds, call `ServerState::with_routing_log(path)` in [`crates/switchyard-server/src/lib.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/lib.rs). This creates a `SharedRoutingLog` holding an `Arc<Mutex<RoutingLog>>`:

```rust
use switchyard_server::ServerState;
use std::path::Path;

let server = ServerState::new(config)
    .with_routing_log(Path::new("/var/log/switchyard/routing.jsonl"))?;

```

The constructor ensures the target directory exists and opens the file in append-only mode using `fs::File` with `create(true).append(true)` flags, guaranteeing durability even if the process crashes.

## Understanding the Routing Log Structure

Each line in the routing log represents a `RoutingRecord` struct serialized as JSON. The logging pipeline in [`crates/switchyard-server/src/routing_log.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/routing_log.rs) follows three stages:

### Stage 1: Context Extraction

For every incoming request, the server builds a `RoutingLogContext` from HTTP headers and the `Metadata` object. This extracts:

- `task` — optional workflow identifier
- `trial_id` — A/B test or experiment identifier
- `session_id` — primary grouping key (falls back to `proxy_x_session_id` header for backward compatibility)

The implementation at lines 99–124 handles header parsing with graceful degradation when optional fields are absent.

### Stage 2: Record Appending

When a request completes, the server calls `SharedRoutingLog::append(context, model, tier, usage)`. This method:

1. Acquires the mutex lock on the underlying `RoutingLog`
2. Serializes the `RoutingRecord` to JSON
3. Writes the line with a trailing newline
4. Releases the lock

Lock acquisition failures are logged but do not abort the request, ensuring the server remains available even under logging pressure.

### Stage 3: Snapshot Retrieval

The `routing_log::snapshot(path, session_id)` function provides read-side analysis. It lazily scans the JSONL file line-by-line at lines 73–95, deserializing each record and aggregating statistics only for the specified session. Malformed lines trigger `continue` rather than panic, making the reader tolerant of partially-written records during active server operation.

## Identifying Routing Failures in Log Entries

Failed routing attempts manifest through specific field patterns in the JSON records:

| Failure Pattern | Indicator Field | Interpretation |
|-----------------|-----------------|--------------|
| **No available model** | `"model": ""` or missing | Router found no capable backend for the request |
| **Cache miss with fallback** | `"fallback_reason": "cache_miss"` | Request eligible for caching but no hit; routed to alternative |
| **Tier exhaustion** | `"tier": "fallback"` with zero tokens | Primary tier saturated; emergency routing activated |
| **Configuration error** | `null` routing fields | Route table lookup failed or returned invalid entry |

Use `jq` for ad-hoc filtering:

```bash

# Find all routing failures for a specific session

jq 'select(.session_id=="abc123" and .model=="")' /var/log/switchyard/routing.jsonl

# Find fallback reasons across all sessions

jq 'select(.fallback_reason!=null) | {session_id, fallback_reason, timestamp}' /var/log/switchyard/routing.jsonl

```

## Extracting Session Snapshots for Deep Analysis

For programmatic debugging, use the snapshot API to aggregate token statistics and call patterns per session:

```rust
use switchyard_server::routing_log::{snapshot, SessionStatsSnapshot};
use std::path::Path;

fn diagnose_routing(path: &Path, session_id: &str) {
    match snapshot(path, session_id) {
        Ok(Some(stats)) => {
            println!("Total calls: {}", stats.total_calls);
            println!("Prompt tokens: {}", stats.total_prompt_tokens);
            println!("Completion tokens: {}", stats.total_completion_tokens);
            
            // Per-model breakdown reveals routing distribution
            for (model, count) in &stats.per_model_calls {
                println!("  {}: {} calls", model, count);
            }
        }
        Ok(None) => eprintln!("No records found for session {}", session_id),
        Err(e) => eprintln!("Log read error: {}", e),
    }
}

```

The `SessionStatsSnapshot` returned by `snapshot` includes:

- `total_calls` — number of requests in session
- `total_prompt_tokens` / `total_completion_tokens` — aggregated token usage
- `per_model_calls` — `HashMap<String, usize>` showing which models served the session

An empty or unexpectedly small `per_model_calls` map with high `total_calls` indicates repeated routing failures where no model could be assigned.

## Python Analysis for Operations Teams

For teams preferring Python for log analysis:

```python
import json
from collections import defaultdict
from dataclasses import dataclass
from typing import Optional


@dataclass
class RoutingStats:
    total_calls: int = 0
    prompt_tokens: int = 0
    completion_tokens: int = 0
    model_calls: dict = None
    failures: list = None
    
    def __post_init__(self):
        if self.model_calls is None:
            self.model_calls = defaultdict(int)
        if self.failures is None:
            self.failures = []


def analyze_session(log_path: str, session_id: str) -> RoutingStats:
    stats = RoutingStats()
    
    with open(log_path, "r") as f:
        for line_num, line in enumerate(f, 1):
            try:
                rec = json.loads(line)
                if rec.get("session_id") != session_id:
                    continue
                
                stats.total_calls += 1
                stats.prompt_tokens += rec.get("prompt_tokens", 0)
                stats.completion_tokens += rec.get("completion_tokens", 0)
                
                model = rec.get("model", "")
                if model:
                    stats.model_calls[model] += 1
                else:
                    # Capture routing failure with context

                    stats.failures.append({
                        "line": line_num,
                        "timestamp": rec.get("timestamp"),
                        "fallback_reason": rec.get("fallback_reason"),
                        "tier": rec.get("tier")
                    })
                    
            except json.JSONDecodeError:
                continue  # Mirror Rust implementation's graceful handling

    
    return stats


# Usage

stats = analyze_session("/var/log/switchyard/routing.jsonl", "abc123")
print(f"Success rate: {(stats.total_calls - len(stats.failures)) / stats.total_calls:.1%}")
for failure in stats.failures:
    print(f"Routing failure at line {failure['line']}: {failure.get('fallback_reason', 'unknown')}")

```

## Correlating Logs with Runtime Metrics

Routing logs provide decision-level detail, but root-cause analysis often requires combining with the server's Prometheus metrics at `/metrics`. Key correlations:

- **High latency + empty model field** — indicates timeout during backend discovery; check `switchyard_backend_health` metric
- **Fallback reason present + cache hit rate drop** — verify `switchyard_cache_hits_total` decreased for the relevant tier
- **Token count mismatch** — compare `usage.prompt_tokens` in log with `switchyard_tokens_processed_total` to detect accounting errors

## Summary

- **Enable routing logs** via `--routing-log <PATH>` CLI flag or `ServerState::with_routing_log()` programmatically
- **Inspect failure indicators** including empty `model` fields, `fallback_reason` keys, and `tier="fallback"` values
- **Use `snapshot()` API** to aggregate session-level statistics without parsing the entire log file
- **Leverage append-only safety** — read and analyze logs while the server runs without synchronization overhead
- **Combine with Prometheus** — correlate routing decisions with backend health and cache metrics for complete diagnosis

## Frequently Asked Questions

### What file format does Switchyard-Server use for routing logs?

Switchyard-Server writes **line-delimited JSON (JSONL)** files. Each line is an independent `RoutingRecord` struct serialized with `serde_json`, making the format compatible with standard Unix tools like `jq`, `grep`, and line-oriented streaming parsers. The append-only design ensures readers never need file locking.

### Can I rotate or delete routing logs while the server is running?

Yes. The `RoutingLog` implementation in [`crates/switchyard-server/src/routing_log.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/crates/switchyard-server/src/routing_log.rs) opens the file once at initialization and writes through the retained `fs::File` descriptor. External log rotation (via `logrotate` with `copytruncate` or similar) will not crash the server, though the new file descriptor will not be automatically picked up without a restart.

### Why does my session show zero token counts despite successful requests?

Zero token counts in `RoutingRecord` entries indicate the request completed routing but usage metrics were unavailable. This typically occurs when: (1) the backend returned a non-200 status without usage headers, (2) the request was rejected before tokenization, or (3) the [`usage_metrics.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/usage_metrics.rs) integration failed to populate the `Metadata` object. Check adjacent log entries for `fallback_reason` fields to distinguish these cases.

### How do I debug a routing failure when no session_id is provided?

When `session_id` is absent from headers, `RoutingLogContext` falls back to the legacy `proxy_x_session_id` header as implemented at lines 99–124. If neither is present, the field will be `null` in the JSON record. Filter logs by `task` or `trial_id` instead, or configure your load balancer to inject identifiable headers before requests reach Switchyard-Server.