How to Set Up Prometheus Metrics Collection in Switchyard Server: Complete Configuration Guide

Enable Prometheus metrics in Switchyard by passing the --metrics-bind flag when starting the server, which exposes a /metrics endpoint on a configurable port for scraper access.

Switchyard, NVIDIA's modular ML serving framework, includes a native Prometheus exporter built directly into the switchyard-server binary. This integration eliminates the need for external exporters or sidecar containers—you simply enable the metrics endpoint via command-line flag or environment variable, and the server handles the rest. The implementation leverages the Rust prometheus crate to collect and serve operational telemetry with minimal overhead.

Understanding Switchyard's Metrics Architecture

The metrics system resides in the switchyard-server crate and follows a registry-based pattern common to Prometheus client libraries. When enabled, the server spawns an independent async HTTP listener that exclusively serves the /metrics endpoint. This separation ensures that metrics scraping never interferes with the main inference API traffic.

According to the NVIDIA-NeMo/Switchyard source code, the key components are:

• crates/switchyard-server/src/main.rs — Parses CLI arguments including --metrics-bind, validates the configuration, and initializes the metrics subsystem before the main server loop starts.
• crates/switchyard-server/src/metrics.rs — Creates the Prometheus Registry, defines metric families (counters, histograms, gauges), and runs the HTTP handler that serializes registry state into the Prometheus exposition format.
• crates/switchyard-server/Cargo.toml — Declares the prometheus crate dependency with default features enabled for text format encoding.

Enabling the Metrics Endpoint

Switchyard ships with metrics disabled by default. You have two equivalent methods to activate the exporter:

Method 1: Command-Line Flag

Pass --metrics-bind with a socket address:

switchyard-server \
  --config routes.toml \
  --port 4000 \
  --metrics-bind 0.0.0.0:9090

This starts the metrics server on all interfaces at port 9090. The endpoint responds at http://0.0.0.0:9090/metrics.

Method 2: Environment Variable

Set SWITCHYARD_METRICS_BIND before launching:

export SWITCHYARD_METRICS_BIND="0.0.0.0:9090"
switchyard-server --config routes.toml --port 4000

Environment variables take precedence over CLI defaults but can be overridden by explicit flags.

Optional: Customizing the Metrics Path

To serve metrics at a non-standard path, export SWITCHYARD_METRICS_PATH:

export SWITCHYARD_METRICS_PATH="/custom/metrics"

The default value is /metrics when unspecified.

Configuring Prometheus to Scrape Switchyard

Add a scrape job to your prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'switchyard_server'
    static_configs:
      - targets: ['localhost:9090']
    metrics_path: '/metrics'
    scrape_interval: 10s

For Kubernetes deployments, use service discovery:

scrape_configs:
  - job_name: 'switchyard_k8s'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: switchyard-server
      - source_labels: [__address__]
        target_label: __address__
        replacement: ${1}:9090

Metrics Available

The Switchyard exporter exposes standard server-side telemetry. Based on the implementation in crates/switchyard-server/src/metrics.rs, expect these metric families:

Metric Name	Type	Description
switchyard_requests_total	Counter	Total requests processed since server start
switchyard_request_latency_seconds	Histogram	Request duration distribution across configurable buckets
switchyard_active_connections	Gauge	Currently open client connections
switchyard_errors_total	Counter	Cumulative error responses by category

All metrics include default labels for the server instance and, where applicable, route identifiers from your routes.toml configuration.

Verification and Troubleshooting

Quick Health Check

# Test metrics endpoint directly

curl http://localhost:9090/metrics

# Verify specific metric presence

curl -s http://localhost:9090/metrics | grep switchyard_requests_total

Common Issues

• Connection refused: Confirm the --metrics-bind flag was passed and the port is not blocked by firewall rules. Check netstat -tlnp for the listening process.
• Empty response: The metrics module initializes lazily—send a few requests through the main API to populate counters before scraping.
• Port conflicts: Switchyard binds the metrics listener separately from the main API port; ensure 9090 (or your chosen port) is available.

Debugging Startup

Launch with verbose logging to confirm metrics initialization:

RUST_LOG=switchyard_server=debug switchyard-server --metrics-bind 0.0.0.0:9090

Look for log lines containing metrics::init and prometheus registry to verify successful setup.

Performance Considerations

The Switchyard metrics implementation uses lock-free data structures where possible and serializes the registry only during scrape requests. Observed overhead in production deployments is typically <1% of CPU under moderate load. For high-throughput scenarios:

• Increase scrape_interval in Prometheus to reduce serialization frequency
• Enable metric aggregation via recording rules rather than raw histogram queries
• Consider the --metrics-buffer-size flag (if available in your version) to tune response buffering

Summary

• Switchyard server includes a built-in Prometheus exporter in crates/switchyard-server/src/metrics.rs—no external components required
• Enable with --metrics-bind <address> or SWITCHYARD_METRICS_BIND environment variable
• Default endpoint is /metrics but customizable via SWITCHYARD_METRICS_PATH
• Source files: main.rs for CLI handling, metrics.rs for registry and HTTP server, Cargo.toml for crate dependencies

Frequently Asked Questions

What port should I use for Switchyard Prometheus metrics?

Use any available TCP port that your Prometheus installation can reach. Port 9090 is conventional and matches Prometheus's own default, but Switchyard accepts any valid socket address. Ensure the port is exposed in your container or firewall configuration.

Can I disable specific metrics to reduce overhead?

The current Switchyard implementation registers a fixed set of metric families during initialization. Granular disabling requires code modification in crates/switchyard-server/src/metrics.rs to conditionally register collectors based on configuration flags.

Does Switchyard support the Prometheus remote write protocol?

No. Switchyard's native exporter serves the Prometheus text format over HTTP for scraper pull-based collection. For remote write functionality, deploy a Prometheus instance as a sidecar that scrapes Switchyard and forwards via remote write to your backend.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}