Monitoring Connection Stats and Path Metrics in iroh: A Complete Guide

Iroh provides a built-in, lock-free metrics collection system that exposes real-time counters for socket traffic, connection lifecycle events, and path discovery through the Endpoint::metrics() API.

Monitoring connection stats and path metrics in iroh is essential for debugging network performance and understanding how your application utilizes direct versus relayed connections. The library ships with a comprehensive instrumentation layer located in the iroh::metrics module that automatically tracks bytes sent over IPv4, IPv6, and relay paths, as well as hole-punching attempts and connection state changes. These metrics are maintained in memory using monotonic counters and are exposed via a public, zero-cost API on the Endpoint struct.

Architecture of the Metrics System

The monitoring stack is split between socket-level telemetry and endpoint-level aggregation, both implemented using the iroh_metrics crate's Counter type and #[metrics] macro.

Socket-Level Metrics (Metrics)

The core counters reside in iroh/src/socket/metrics.rs within the Metrics struct. This structure tracks low-level network activity including:

• Traffic counters: send_ipv4, send_ipv6, send_relay, and corresponding receive counters
• Path changes: transport_ip_paths_added, transport_relay_paths_removed
• Hole-punching: holepunch_attempts and success indicators
• Actor loop statistics: actor_tick_main, actor_tick_msg, actor_tick_re_stun

These counters implement the MetricsGroup trait and are wrapped in Arc for thread-safe sharing across the socket's internal transports.

Endpoint-Level Aggregation (EndpointMetrics)

Defined in iroh/src/metrics.rs, EndpointMetrics wraps the socket-level Metrics and adds connection-specific counters such as num_conns_opened, num_conns_closed, paths_direct, paths_relay, and paths_custom. The Endpoint::metrics() method in iroh/src/endpoint.rs returns a reference to this struct, allowing applications to query aggregated statistics at any time.

Transport Layer Integration

Each transport implementation—IP (iroh/src/socket/transports/ip.rs), Relay (iroh/src/socket/transports/relay/actor.rs), and custom transports—receives a shared Arc<Metrics> instance. When a transport sends or receives data, it increments the relevant counter directly in the hot path (e.g., self.metrics.send_ipv4.inc_by(bytes)), ensuring metrics reflect real-time activity without locking overhead.

How Metrics Are Collected

Collection happens automatically during normal endpoint operation through five primary stages:

1. Endpoint Initialization: When Endpoint::builder().bind_default().await succeeds, the constructor initializes EndpointMetrics::default(), creating fresh counters for the socket's lifecycle.
2. Data Plane Events: The IP transport increments send_ipv4 or send_ipv6 for every datagram sent, while the Relay actor updates send_relay when traffic flows through fallback servers.
3. Connection Lifecycle: The socket actor increments num_conns_opened only after a successful TLS handshake completes, and increments num_conns_closed when the connection terminates.
4. Path Discovery: As the endpoint discovers direct paths or falls back to relays, it updates paths_direct, paths_relay, and holepunch_attempts counters.
5. Actor Loop Telemetry: The main async loop in the relay actor periodically increments actor_tick_main, actor_tick_msg, and timer-related counters to expose internal scheduling health.

Querying Connection Metrics

Access metrics by calling Endpoint::metrics(), which returns a snapshot of all current counters. Because the underlying Counter type uses atomics, you can safely read metrics from multiple threads without blocking the endpoint's event loop.

use iroh::Endpoint;
use std::time::Duration;
use tokio::time;

// Build an endpoint (metrics are enabled by default)
let endpoint = Endpoint::builder()
    .bind_default()
    .await
    .expect("failed to start iroh endpoint");

// Spawn a task to periodically log statistics
tokio::spawn(async move {
    loop {
        let metrics = endpoint.metrics();
        
        // Traffic statistics
        println!("IPv4 sent: {} bytes", metrics.socket.send_ipv4.get());
        println!("IPv6 sent: {} bytes", metrics.socket.send_ipv6.get());
        println!("Relay sent: {} bytes", metrics.socket.send_relay.get());
        
        // Connection lifecycle
        println!("Connections opened: {}", metrics.num_conns_opened.get());
        println!("Connections closed: {}", metrics.num_conns_closed.get());
        
        // Path discovery
        println!("Direct paths: {}", metrics.paths_direct.get());
        println!("Relay paths: {}", metrics.paths_relay.get());
        
        time::sleep(Duration::from_secs(10)).await;
    }
});

Exporting Metrics to External Systems

Because EndpointMetrics implements serde::Serialize, you can serialize snapshots to JSON for ingestion by Prometheus, Grafana, or custom observability pipelines.

use serde_json::json;

let snapshot = endpoint.metrics();
let payload = json!({
    "send_ipv4": snapshot.socket.send_ipv4.get(),
    "send_ipv6": snapshot.socket.send_ipv6.get(),
    "send_relay": snapshot.socket.send_relay.get(),
    "connections_opened": snapshot.num_conns_opened.get(),
    "connections_closed": snapshot.num_conns_closed.get(),
    "direct_paths": snapshot.paths_direct.get(),
    "relay_paths": snapshot.paths_relay.get(),
});

println!("{}", serde_json::to_string_pretty(&payload).unwrap());

Integrating with Tracing

Iroh emits internal tracing events for certain metric thresholds. To capture these, initialize a subscriber before starting the endpoint:

use tracing_subscriber::FmtSubscriber;

let subscriber = FmtSubscriber::builder()
    .with_max_level(tracing::Level::INFO)
    .finish();
tracing::subscriber::set_global_default(subscriber)
    .expect("setting default subscriber failed");

This surfaces diagnostic events emitted by the socket actor and transport layers without requiring manual metric polling.

Summary

• Lock-free counters: Iroh uses the iroh_metrics crate's Counter type for zero-overhead, atomic metric collection across threads.
• Two-tier architecture: Metrics tracks socket-level traffic in iroh/src/socket/metrics.rs, while EndpointMetrics aggregates connection state in iroh/src/metrics.rs.
• Public API: Call Endpoint::metrics() to retrieve a serializable snapshot of all counters, including bytes sent via IPv4, IPv6, and relay paths.
• Automatic collection: Transports increment counters directly during send/receive operations, ensuring real-time accuracy without manual instrumentation.
• Serialization ready: The metrics struct implements serde::Serialize, enabling easy export to JSON or binary formats like postcard for remote diagnostics.

Frequently Asked Questions

How do I access connection metrics in iroh?

Call the Endpoint::metrics() method on your endpoint instance. This returns an EndpointMetrics struct containing socket-level traffic counters (send_ipv4, send_relay) and connection lifecycle counters (num_conns_opened, paths_direct). Access individual values using the .get() method on each Counter field.

What is the performance impact of enabling metrics?

The impact is negligible. Metrics are enabled by default and use lock-free atomic counters from the iroh_metrics crate. Increments occur via Arc<Metrics> shared across threads, and reads through Endpoint::metrics() do not block the socket's async event loop.

Can I export iroh metrics to Prometheus?

Yes. Since EndpointMetrics implements serde::Serialize, you can serialize the struct to JSON and convert it to Prometheus exposition format. Poll endpoint.metrics() periodically in your application code and expose the values via your existing metrics endpoint.

What is the difference between socket-level and endpoint-level metrics?

Socket-level metrics (defined in iroh/src/socket/metrics.rs) track raw network activity such as bytes sent and path changes. Endpoint-level metrics (defined in iroh/src/metrics.rs) aggregate these socket counters and add higher-level connection state tracking, such as the total number of opened connections and discovered direct versus relay paths.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}