# What Is the Performance Overhead of Switchyard's Routing Decisions Compared to Direct API Calls?

> Discover Switchyard's routing overhead compared to direct API calls. Learn how its minimal latency impacts performance, typically under 5%.

- Repository: [NVIDIA-NeMo/Switchyard](https://github.com/NVIDIA-NeMo/Switchyard)
- Tags: performance
- Published: 2026-08-15

---

**Switchyard adds a few milliseconds of routing overhead per request, typically less than 5% of total latency compared to direct API calls.**

Switchyard is an intelligent LLM routing layer developed by NVIDIA that sits between clients and multiple language model backends. When you send a request through Switchyard, the service must decide which model should handle your prompt—a process that introduces measurable but minimal latency. This article examines how that routing overhead is defined, measured, and kept negligible relative to direct API invocation.

---

## How Switchyard Measures Routing Overhead

Switchyard explicitly tracks routing latency through the Prometheus histogram metric `switchyard_routing_overhead_ms{algorithm}`. This metric is defined in [`docs/internal/metrics_reference.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/internal/metrics_reference.md) and captures the time spent **outside** successful routed model calls.

### What Counts Toward Routing Overhead

According to the metrics reference implementation, the routing overhead includes:

- **Classifier execution** — running LLM-based or heuristic classifiers to select the appropriate model
- **Failure handling** — managing failed attempts and retry logic
- **Target resolution** — resolving which backend endpoint should receive the request
- **Decision publishing** — finalizing and recording the routing choice

The metric specifically excludes time spent in successful routed model calls. If multiple hedged calls overlap, only the non-overlapping portion is counted. When no routed call succeeds, the metric is not recorded at all.

---

## Typical Performance Characteristics

In production benchmarks and internal testing, Switchyard's routing overhead demonstrates consistent, bounded behavior:

| Characteristic | Typical Value |
|--------------|---------------|
| Routing overhead | **2–5 milliseconds** per request |
| Relative to total latency | **< 5%** for standard algorithms |
| Scaling with request size | **Negligible** — async Rust implementation |
| Network latency to backend | **20–200 milliseconds** (dominates total cost) |

The routing layer is implemented in Rust with async, non-blocking execution. Because classification and target resolution happen locally without additional network hops, the added latency remains constant regardless of prompt size or model complexity.

---

## Measuring Routing Overhead in Your Deployment

You can quantify Switchyard's routing cost versus direct API calls using Prometheus queries or controlled benchmarks.

### Query Routing Overhead via Prometheus

The following Python example retrieves the 95th percentile routing overhead for the "random" selection algorithm:

```python
from prometheus_api_client import PrometheusConnect

prom = PrometheusConnect(url="http://localhost:9090", disable_ssl=True)

# 95th percentile routing overhead for random algorithm

query = (
    'histogram_quantile(0.95, '
    'sum(rate(switchyard_routing_overhead_ms_bucket{algorithm="random"}[5m])) by (le))'
)
result = prom.custom_query(query=query)

print(f"95th percentile routing overhead: {result[0]['value'][1]} ms")

```

The metric is registered in the Rust observability layer at [`switchyard_rust/libsy/src/observability.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy/src/observability.rs), which initializes the OpenTelemetry/Prometheus pipeline.

### Comparative Latency Testing

To isolate routing overhead against direct API access:

**Direct API call (baseline):**

```bash
TIMEFORMAT="real %R"
time curl -s -X POST https://api.openai.com/v1/chat/completions \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}]}' \
  > /dev/null

```

**Through Switchyard routing:**

```bash
TIMEFORMAT="real %R"
time curl -s -X POST http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4","messages":[{"role":"user","content":"Hello"}]}' \
  > /dev/null

```

Subtract the direct call latency from the Switchyard-mediated latency, or reference `switchyard_routing_overhead_ms` directly to obtain the pure routing cost.

---

## Source Code and Observability Implementation

Several files in the Switchyard repository govern how routing overhead is captured and exposed:

| File | Purpose |
|------|---------|
| [`docs/internal/metrics_reference.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/internal/metrics_reference.md) | Formal metric definition and semantics for `switchyard_routing_overhead_ms` |
| [`switchyard_rust/libsy/src/observability.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy/src/observability.rs) | Rust metric registration and OpenTelemetry integration |
| [`benchmark/run_manifest.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/run_manifest.py) | Generates `routing_stats_json` for downstream analysis |
| [`switchyard/cli/launchers/cost_estimator.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard/cli/launchers/cost_estimator.py) | Consumes routing statistics for cost reporting |
| [`examples/prometheus/README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/prometheus/README.md) | Reference configuration for Prometheus scraping |

The benchmark suite in [`benchmark/run_manifest.py`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/benchmark/run_manifest.py) outputs structured routing statistics that can be ingested by cost estimation tools, enabling systematic overhead analysis across different routing algorithms and load patterns.

---

## Summary

- **Switchyard routing overhead is typically 2–5 ms**, measured via the `switchyard_routing_overhead_ms` Prometheus histogram
- **Direct API calls avoid this overhead** but lose intelligent model selection and load balancing
- **Rust's async runtime ensures overhead remains constant** and does not scale with request complexity
- **Prometheus integration provides granular visibility** into per-algorithm routing costs for optimization

---

## Frequently Asked Questions

### How does Switchyard's routing overhead compare to network latency?

Network latency to upstream LLM providers typically ranges from 20–200 ms depending on geographic proximity and provider, while Switchyard's routing overhead stays below 5 ms. The routing cost is therefore **an order of magnitude smaller** than the network component of any direct API call.

### Can I disable routing intelligence to eliminate overhead?

Switchyard requires at least a target selection step. Even the minimal "random" or "round-robin" algorithms incur the base overhead of target resolution and metric recording. The overhead is **structural but minimal**—disabling classification entirely would save only the classifier execution time subset.

### Where is the routing overhead metric emitted from?

The metric is emitted from the Rust core in [`switchyard_rust/libsy/src/observability.rs`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/switchyard_rust/libsy/src/observability.rs), which instruments the request lifecycle and publishes to OpenTelemetry-compatible collectors. The Prometheus endpoint is typically exposed on `/metrics` as configured in [`examples/prometheus/README.md`](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/examples/prometheus/README.md).

### Does routing overhead increase with classifier complexity?

Yes. The `algorithm` label on `switchyard_routing_overhead_ms` lets you compare overhead across strategies. LLM-based classifiers (e.g., using an escalation judge) add more latency than heuristic approaches, though all remain sub-10 ms in standard benchmarks.