How to Implement Custom Authentication Logic in Iroh: 3 Proven Methods

To implement custom authentication logic in Iroh, extend the handshake protocol in iroh-relay/src/protos/handshake.rs or implement the AccessControl trait from iroh-relay/src/server.rs to enforce fine-grained authorization policies after cryptographic verification.

Iroh's relay layer authenticates clients during the TLS or QUIC handshake and authorizes them before allowing traffic. The n0-computer/iroh repository provides two primary extension points for customizing this behavior: replacing the default challenge-response handshake or implementing post-authentication access controls. This guide covers three methods to implement custom authentication logic in Iroh, from low-level protocol changes to high-level policy enforcement.

Understanding Iroh's Authentication Architecture

Iroh separates identity verification (authentication) from permission grants (authorization). The default flow uses cryptographic challenge-response to prove identity, then checks permissions via a pluggable trait.

The Handshake Protocol

In iroh-relay/src/protos/handshake.rs, the server handles two authentication mechanisms:

• Challenge-Response: The server generates a random challenge that the client signs with the secret key belonging to its EndpointId. The server verifies this using ClientAuth::verify (lines 456-470).
• Key-Material Authentication: If the client sends a KeyMaterialClientAuth header, the server validates TLS keying material instead of the challenge (lines 425-444).

After successful cryptographic verification, the server creates a SuccessfulAuthentication instance and calls authorize_with, which delegates to the AccessControl trait implementation (defined in iroh-relay/src/server.rs, line 285).

The Access Control Layer

Authentication proves who the client is. Authorization decides what they may do. The AccessControl trait provides the hook to implement custom authorization logic after the cryptographic handshake succeeds.

Method 1: Extend the Handshake Protocol

Replace or extend the default handshake to add custom authentication mechanisms like device certificates or hardware security modules.

Hook location: handshake::serverside constructs the SuccessfulAuthentication instance after verifying the client auth header (lines 418-442 in iroh-relay/src/protos/handshake.rs).

Implementation steps:

1. Create a new struct (e.g., CustomDeviceAuth) implementing serde::Deserialize and a verify method:

use serde::Deserialize;

#[derive(Deserialize)]
struct CustomDeviceAuth {
    device_cert: Vec<u8>,
    signature: Vec<u8>,
}

impl CustomDeviceAuth {
    async fn verify(&self, io: &mut impl AsyncRead + AsyncWrite) -> anyhow::Result<()> {
        // Verify hardware-backed device certificate
        // Return Ok(()) if valid, Err otherwise
        todo!()
    }
}

2. Modify serverside to attempt deserialization into your struct before falling back to ClientAuth:

// In iroh-relay/src/protos/handshake.rs serverside function
if let Ok(custom_auth) = postcard::from_bytes::<CustomDeviceAuth>(&client_auth_header) {
    custom_auth.verify(&mut io).await?;
    // Extract public key from custom_auth
    return Ok(SuccessfulAuthentication::new(public_key));
}
// Fall back to default ClientAuth

3. Return a SuccessfulAuthentication containing the extracted public key.

Method 2: Implement Custom AccessControl

Implement fine-grained authorization policies without changing the cryptographic handshake. This method validates JWTs, queries databases, or checks allow-lists after the client proves possession of their private key.

The AccessControl trait resides in iroh-relay/src/server.rs (lines 285-306):

pub trait AccessControl: Send + Sync + 'static {
    fn authorize(
        &self,
        client_key: &PublicKey,
        request: &ClientRequest,
    ) -> Result<(), Error>;
}

Implementation example:

use iroh_base::key::PublicKey;
use iroh_relay::server::{AccessControl, ClientRequest, Error};
use std::collections::HashSet;

#[derive(Clone)]
struct DatabaseAccessControl {
    allowed_keys: HashSet<PublicKey>,
    // Add DB pool or API client here
}

impl AccessControl for DatabaseAccessControl {
    fn authorize(
        &self,
        client_key: &PublicKey,
        request: &ClientRequest,
    ) -> Result<(), Error> {
        // Check if key exists in organization database
        if !self.allowed_keys.contains(client_key) {
            return Err(Error::AccessDenied("Key not in organization".into()));
        }
        
        // Additional checks: validate JWT in request.auth_token(), etc.
        Ok(())
    }
}

Wire it into the server using ServerBuilder (from iroh-relay/src/server.rs):

use iroh_relay::server::ServerBuilder;
use std::sync::Arc;

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let allowed_keys = load_keys_from_database().await?;
    let access = Arc::new(DatabaseAccessControl { allowed_keys });
    
    let server = ServerBuilder::new()
        .with_access_control(access)  // Inject custom logic
        .bind(([0, 0, 0, 0], 443))
        .await?;
        
    server.run().await?;
    Ok(())
}

Method 3: Static Token-Based Authentication (Shortcut)

For simple use cases requiring a shared secret across all clients, use the built-in static token support in iroh-relay/src/relay_map.rs (lines 266-277).

Configure the client with RelayConfig::with_auth_token:

use iroh_relay::relay_map::RelayConfig;

let config = RelayConfig::new(url)
    .with_auth_token("secret-token-123".to_string());

The server extracts this token via Server::auth_token() (lines 252-264 in iroh-relay/src/server.rs). Combine this with a custom AccessControl implementation to validate the token against an external authority:

impl AccessControl for TokenValidator {
    fn authorize(&self, client_key: &PublicKey, request: &ClientRequest) -> Result<(), Error> {
        if let Some(token) = request.auth_token() {
            if validate_against_oauth(token)? {
                return Ok(());
            }
        }
        Err(Error::AccessDenied("Invalid token".into()))
    }
}

Complete Minimal Example

This example demonstrates custom AccessControl with an in-memory allow-list:

use iroh_base::key::PublicKey;
use iroh_relay::server::{ServerBuilder, AccessControl, ClientRequest, Error};
use std::collections::HashSet;
use std::sync::Arc;

#[derive(Clone)]
struct AllowList {
    allowed: HashSet<PublicKey>,
}

impl AccessControl for AllowList {
    fn authorize(&self, client_key: &PublicKey, _req: &ClientRequest) -> Result<(), Error> {
        if self.allowed.contains(client_key) {
            Ok(())
        } else {
            Err(Error::AccessDenied("unknown client".into()))
        }
    }
}

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut allowed = HashSet::new();
    allowed.insert(PublicKey::from_hex("0123…")?);
    
    let access = Arc::new(AllowList { allowed });
    let server = ServerBuilder::new()
        .with_access_control(access)
        .bind(([0, 0, 0, 0], 0))
        .await?;
        
    server.run().await?;
    Ok(())
}

Summary

• Handshake extension: Modify iroh-relay/src/protos/handshake.rs to replace ClientAuth with custom challenge-response mechanisms or certificate validation.
• AccessControl traits: Implement the AccessControl trait from iroh-relay/src/server.rs to enforce policies after cryptographic authentication, rejecting clients based on external databases, OAuth tokens, or IP restrictions.
• Static tokens: Use RelayConfig::with_auth_token for simple shared-secret scenarios, validating the token inside your AccessControl implementation.
• Server construction: Pass custom implementations via ServerBuilder::with_access_control to inject your logic into the relay server.

Frequently Asked Questions

What is the difference between authentication and authorization in Iroh?

Authentication verifies that a client possesses the private key corresponding to their public identity, typically through the challenge-response handshake in iroh-relay/src/protos/handshake.rs. Authorization occurs afterward via the AccessControl trait, determining whether that authenticated identity has permission to use the relay.

Can I use JWT or OAuth tokens with Iroh's relay server?

Yes. Implement the AccessControl trait and extract the token from ClientRequest::auth_token() within the authorize method. You can then validate the JWT against your identity provider or OAuth server before returning Ok(()) or Error::AccessDenied.

Do I need to modify client code when implementing custom server authentication?

If you extend the handshake protocol in protos/handshake.rs, you must update the client to send your custom authentication header. If you only implement custom AccessControl or use static tokens via RelayConfig, standard Iroh clients work without modification.

Where should I store allowed public keys for my AccessControl implementation?

Store them in any external system accessible from your AccessControl implementation, such as a PostgreSQL database, Redis cache, or LDAP directory. Initialize your struct with connection pools in main() and pass it to ServerBuilder::with_access_control.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}