Debugging Connection Issues with iroh's Structured Events and Logging

Iroh uses the tracing crate to emit structured events under the iroh::_events target hierarchy, enabling you to diagnose connection failures by filtering logs for path state changes, relay connections, and link events.

Debugging connection issues with iroh's structured events and logging requires understanding how the library emits structured diagnostics via the tracing crate. The n0-computer/iroh repository instruments its QUIC socket layer with granular events that reveal why paths are abandoned, why hole-punching fails, or why relays are unreachable. These structured events are emitted using the tracing::event! macro rather than standard debug! calls, allowing you to attach arbitrary key-value pairs that can be filtered or inspected by log collectors.

Where Structured Events Are Emitted

Iroh emits structured events from several key components in the socket layer. Each event uses a specific target string under the iroh::_events namespace that you can filter in your logging configuration.

The primary event sources are:

• iroh::_events::path::open – Emitted in iroh/src/socket/remote_map/remote_state.rs within the RemoteStateActor::handle_path_event method when a new QUIC path is opened.
• iroh::_events::path::selected – Emitted when the path selector chooses a new preferred path for a remote endpoint.
• iroh::_events::path::abandoned – Emitted when a path is dropped because no connection uses it anymore.
• iroh::_events::relay::connected – Emitted in iroh/src/socket/transports/relay/actor.rs when the relay actor successfully connects to a relay server.
• iroh::_events::link_change – Emitted when a local network interface is added or removed.

These events are emitted using the event! macro with explicit targets, allowing you to attach structured fields such as remote = %self.state.endpoint_id.fmt_short(). The handle_path_event method in remote_state.rs (lines 616-1020) contains the logic for path-related events, while relay events appear in actor.rs around lines 388-511.

How Events Propagate Through the System

The event pipeline follows an actor-based architecture that broadcasts state changes to all subscribers.

PathStateSender owns the broadcast channel and records path state changes. Located in iroh/src/socket/remote_map/remote_state/path_watcher.rs, the record_opened method (lines 69-96) forwards events over a broadcast channel with a fixed capacity of BROADCAST_CAPACITY = 8.

PathStateReceiver is held by each Connection and builds a PathEventStream from the broadcast channel. Consumers can await on this stream to receive live events.

RemoteStateActor consumes a merged stream of all connections' path events and reacts to them, such as selecting new paths or triggering hole-punching. The handle_path_event method contains the match logic on NoqPathEvent (around lines 610-640 in remote_state.rs).

When subscribers cannot keep up with the broadcast channel's capacity, the system emits a PathEvent::Lagged event. This serves as a diagnostic signal that the event pipeline is being saturated.

Enabling and Filtering Logs

Iroh does not hard-code a logger; instead, you must initialize a tracing_subscriber layer in your binary or test. The standard pattern appears in iroh/tests/integration.rs (lines 150-154):

tracing_subscriber::fmt()
    .with_env_filter(tracing_subscriber::EnvFilter::from_default_env())
    .init();

Control verbosity via the RUST_LOG environment variable (or IROH_LOG). To display all path-related events at debug level:

RUST_LOG=iroh::_events=debug cargo run --example monitor-connections

Filter specific targets to isolate issues:

RUST_LOG=iroh::_events::path::selected=info,iroh::_events::relay::connected=debug cargo run

Adding ,trace to the filter includes lower-level trace! calls scattered throughout the packet processing code.

Inspecting Events Programmatically

To consume events inside your application, create a PathEventStream from a connection:

use iroh::endpoint::Connection;
use iroh::socket::remote_map::remote_state::PathEventStream;

async fn watch_path_events(conn: Connection) {
    let mut events: PathEventStream = conn.path_events();
    while let Some(event) = events.next().await {
        match event {
            PathEvent::Opened { id, remote_addr, .. } => {
                println!("opened path {id} to {remote_addr}");
            }
            PathEvent::Selected { id, .. } => {
                println!("selected path {id}");
            }
            PathEvent::Lagged { missed } => {
                eprintln!("lost {missed} path events");
            }
            _ => {}
        }
    }
}

The PathEventStream type is defined in iroh/src/socket/remote_map/remote_state/path_watcher.rs (lines 31-38) and implements Stream<Item = PathEvent>.

Typical Debugging Workflow

Follow these steps to diagnose connection failures:

1. Enable full tracing – Set RUST_LOG=iroh=debug,iroh::_events=debug to see every structured event, including path opens, selections, and failures.
2. Narrow to specific targets – Use RUST_LOG=iroh::_events::path::abandoned=debug to isolate when a path is being dropped.
3. Observe lagged events – Look for PathEvent::Lagged in the stream or trace! messages about event lagged to identify broadcast channel saturation.
4. Correlate with connection IDs – Filter on a single conn_id included in all events to reduce noise.
5. Verify selector decisions – Enable iroh::_events::path::selected and compare the selected address with the expected one.
6. Check relay status – Enable iroh::_events::relay::connected to verify relay connectivity, as missing relay connections often explain why paths cannot be opened.

Common Pitfalls

• Missing tracing_subscriber initialization – Without a subscriber, all event! calls are silently dropped. Ensure you call tracing_subscriber::fmt::init() early in main.
• Over-filtering – Setting only info level globally suppresses debug-level path events. Use RUST_LOG=debug or explicitly set the target level.
• Broadcast overflow – The default capacity of 8 may be too low for high-throughput workloads. When you see PathEvent::Lagged, consider increasing BROADCAST_CAPACITY in path_watcher.rs.

Summary

• Iroh emits structured events via the tracing::event! macro under the iroh::_events hierarchy.
• Path events are emitted in remote_state.rs and propagated via PathStateSender in path_watcher.rs.
• Relay events are emitted in transports/relay/actor.rs.
• Initialize logging with tracing_subscriber::fmt() and control verbosity with RUST_LOG environment variables.
• Use Connection::path_events() to obtain a PathEventStream for programmatic inspection.
• Watch for PathEvent::Lagged to detect event pipeline saturation.

Frequently Asked Questions

Why don't I see any iroh log events?

Ensure you have initialized a tracing_subscriber before creating the iroh endpoint. Without a subscriber, all tracing::event! calls are silently discarded. Call tracing_subscriber::fmt().with_env_filter(EnvFilter::from_default_env()).init() early in your main function.

What does PathEvent::Lagged indicate?

This event indicates that the broadcast channel (with capacity BROADCAST_CAPACITY = 8) is full and your consumer missed events. It signals that the event pipeline is being saturated, which may happen under high-throughput workloads. Consider increasing the buffer capacity or consuming events faster.

How do I filter logs for a specific connection?

All events include a conn_id field. You can filter your logs to show only events for a specific connection by using structured filtering in your log collector, or by searching for the specific connection ID string in the log output. The events include structured fields like remote = %self.state.endpoint_id.fmt_short() that help correlate events.

Where are relay connection events emitted?

Relay connection events are emitted in iroh/src/socket/transports/relay/actor.rs (lines 388-511) using the target iroh::_events::relay::connected. These events fire when the relay actor successfully establishes a connection to a relay server, which is crucial for diagnosing hole-punching failures.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}