INFINI Gateway High Availability and Automatic Failover: Floating-IP Plugin Architecture

INFINI Gateway achieves high availability and automatic failover through a Floating-IP plugin that uses TCP heartbeat monitoring, UDP multicast discovery, and priority-based leader election to coordinate seamless failover between active and standby nodes.

The infinilabs/gateway repository implements a self-contained high availability solution that requires no external coordination services like ZooKeeper or etcd. By leveraging the Floating-IP plugin, the gateway provides INFINI Gateway high availability and automatic failover capabilities through a lightweight state machine built on standard Linux networking primitives and Go's concurrency patterns.

Floating-IP Plugin Architecture

The high availability system centers on a state machine that manages two distinct roles: active (holds the floating IP) and standby (monitors the active node). This logic is implemented primarily in service/floating_ip/floating_ip.go.

State Machine and Initial Role Detection

Upon startup, the StateMachine() function (lines 61-84) determines whether the local node should start as active or standby by probing the configured floating IP address:

client := heartbeat.New()
aliveChan := make(chan bool)
go func() {
    err := client.Start(floatingIPConfig.IP, floatingIPConfig.Echo.EchoPort,
        floatingIPConfig.Echo.EchoDialTimeout, floatingIPConfig.Echo.EchoTimeout,
        func() { aliveChan <- true },   // on success
        func() { aliveChan <- false }) // on failure
    if err != nil { aliveChan <- false }
}()
alive := <-aliveChan

If the heartbeat client successfully connects to an existing echo service on the floating IP, the node assumes standby mode via SwitchToStandbyMode. If the probe fails (indicating no active node is present), the node calls SwitchToActiveMode to assume leadership.

Active Mode Implementation

When a node transitions to active via SwitchToActiveMode (lines 82-129), it executes three critical operations:

1. Atomic state update: Sets the atomicActiveOrNot atomic value to true to signal goroutines to begin maintenance tasks.

2. Network alias configuration: Assigns the floating IP to the local network interface using net.SetupAlias:

   err := net.SetupAlias(floatingIPConfig.Interface,
                         floatingIPConfig.IP, floatingIPConfig.Netmask)

3. Background maintenance goroutines:

   • Gratuitous ARP: Every 10 seconds, arping.GratuitousArpOverIfaceByName broadcasts ARP updates to ensure L2 switches update their MAC tables.
   • Multicast status broadcast: Every 10 seconds, Broadcast(&floatingIPConfig, &req) sends UDP packets containing the node's active status, priority, and echo port to the multicast group.

Both goroutines monitor atomicActiveOrNot and terminate automatically if the node is demoted to standby.

Standby Mode and Automatic Failover

When a node enters SwitchToStandbyMode (lines 80-119), it:

1. Clears the atomic flag to false.
2. Invokes Deactivate(false) to remove the network alias and stop ARP/broadcast goroutines.
3. Launches a watchdog goroutine that continuously monitors the active node:

client.Start(floatingIPConfig.IP, floatingIPConfig.Echo.EchoPort,
            floatingIPConfig.Echo.EchoDialTimeout, floatingIPConfig.Echo.EchoTimeout,
            func() { aliveChan <- true },
            func() { aliveChan <- false })

If the heartbeat client reports a failure (connection timeout or lost acknowledgement), the standby node immediately invokes module.SwitchToActiveMode(), seizing the floating IP and broadcasting its new status. This completes the automatic failover within seconds of detecting a failure.

Heartbeat Protocol and Health Monitoring

The health checking system operates through a custom TCP protocol implemented in service/heartbeat/server.go and service/heartbeat/client.go.

TCP Echo Service

The heartbeat server listens on a configurable port (default 61111) and implements a lightweight binary protocol:

Direction	Byte Value	Meaning
Client → Server	0x01 (Req_REGISTER)	Client registration
Server → Client	0x02 (Res_REGISTER)	Registration ACK
Server → Client	0x03 (Req_HEARTBEAT)	Heartbeat request
Client → Server	0x04 (Res_HEARTBEAT)	Heartbeat response

Heartbeat Client and Failover Triggers

The client (service/heartbeat/client.go) maintains a persistent connection to the active node's echo port. It periodically sends Req_HEARTBEAT messages and expects Res_HEARTBEAT responses within the configured timeout.

When the client detects a timeout or connection reset, it invokes the failure callback registered by the floating_ip module. This callback triggers the standby node's promotion logic, ensuring automatic failover occurs without manual intervention.

Multicast Discovery and Priority-Based Election

To prevent split-brain scenarios and coordinate multiple standby nodes, the system uses UDP multicast discovery defined in service/floating_ip/broadcast.go.

UDP Broadcast Protocol

Nodes broadcast their status via the Request struct (lines 53-58):

type Request struct {
    IsActive   bool
    FloatingIP string
    FixedIP    string
    EchoPort   int
    Priority   int
}

Each node transmits these packets to a configurable multicast address (default 224.3.2.2:7654) every 10 seconds. The Priority field enables deterministic leader election when multiple nodes compete for the active role.

Split-Brain Avoidance

The multicast handler in service/floating_ip/floating_ip.go (ServeMulticastDiscovery, lines 97-138) implements conflict resolution:

• Self-detection: Ignores packets where FixedIP matches the local address.
• Priority-based demotion: If an active node receives a broadcast from a higher-priority node, it voluntarily demotes itself to standby via SwitchToStandbyMode.
• Preemptive takeover: When ForcedSwitchByPriority is enabled, a standby node that detects a lower-priority active node can preemptively seize leadership and rebroadcast its status.

This mechanism ensures that even during network partitions or simultaneous node startups, the cluster converges on a single active node with the highest configured priority.

Key Implementation Files

File	Responsibility
service/floating_ip/floating_ip.go	Core state machine, role transitions, ARP management, and multicast handling
service/floating_ip/broadcast.go	UDP multicast packet construction and transmission
service/heartbeat/server.go	TCP echo service for health checks
service/heartbeat/client.go	Heartbeat client with failure callbacks
gateway.yml	Configuration for floating IP, netmask, interface, priority, and multicast settings

Practical Configuration Examples

Starting the Floating-IP Plugin

// In your main.go or plugin loader
import "infini.sh/gateway/service/floating_ip"

func main() {
    plugin := floating_ip.FloatingIPPlugin{}
    plugin.Setup()   // read config, verify root
    if err := plugin.Start(); err != nil {
        log.Fatalf("failed to start floating_ip: %v", err)
    }
    // Run other gateway components ...
}

Relevant source: service/floating_ip/floating_ip.go – Setup, Start.

Manually Triggering Role Changes

// Force this node to become active
plugin.SwitchToActiveMode()

// Later, demote it back to standby after a short delay
time.Sleep(10 * time.Second)
plugin.SwitchToStandbyMode(0)

Relevant source: SwitchToActiveMode and SwitchToStandbyMode in service/floating_ip/floating_ip.go.

Using the Heartbeat Client Directly

hb := heartbeat.New()
hb.Start(
    "10.0.0.5",                 // target IP (active node)
    61111,                      // echo port
    5* time.Second,             // dial timeout
    10* time.Second,            // read timeout
    func() { log.Println("active node alive") },
    func() { log.Println("active node down – promote!") })

Relevant source: service/heartbeat/client.go.

Sending Custom Multicast Broadcasts

cfg := &floating_ip.FloatingIPConfig{
    BroadcastConfig: config.NetworkConfig{Binding: "224.3.2.2:7654"},
    // other fields omitted for brevity
}
req := &floating_ip.Request{
    IsActive:   true,
    FloatingIP: "192.168.1.100",
    FixedIP:    "192.168.1.10",
    EchoPort:   61111,
    Priority:   900,
}
floating_ip.Broadcast(cfg, req)

Relevant source: service/floating_ip/broadcast.go.

Summary

• INFINI Gateway high availability and automatic failover are implemented through a self-contained Floating-IP plugin that requires no external coordination services.
• The state machine in service/floating_ip/floating_ip.go determines initial roles by probing the heartbeat echo service on the floating IP address.
• Active nodes assume the floating IP using Linux network aliases and gratuitous ARP, while broadcasting their status via UDP multicast every 10 seconds.
• Standby nodes continuously monitor the active node through the heartbeat client in service/heartbeat/client.go, automatically promoting themselves upon detecting a failure.
• Priority-based election and multicast discovery prevent split-brain scenarios, ensuring the highest-priority healthy node always maintains the active role.

Frequently Asked Questions

How does INFINI Gateway detect when the active node has failed?

The standby node runs a heartbeat client (service/heartbeat/client.go) that periodically connects to the active node's TCP echo port (default 61111). If the connection times out or the heartbeat acknowledgement is not received within the configured interval, the client invokes a failure callback. This callback triggers SwitchToActiveMode() in the floating_ip module, causing the standby node to seize the floating IP and assume the active role.

Can multiple standby nodes exist in the same cluster?

Yes. Any number of nodes can operate in standby mode simultaneously. Each standby independently monitors the active node via heartbeats and listens to multicast broadcasts. If the active node fails, all standbys will attempt to promote themselves. However, the priority-based election mechanism ensures that only the node with the highest configured priority successfully transitions to active mode, while others remain in or return to standby.

What prevents two nodes from both claiming the active role simultaneously?

The system implements split-brain avoidance through the multicast discovery handler in service/floating_ip/floating_ip.go. When a node receives a broadcast from another node claiming to be active, it compares priorities. An active node that discovers a higher-priority peer will voluntarily demote itself to standby. Additionally, if ForcedSwitchByPriority is enabled, a standby node can preempt a lower-priority active node. These rules ensure the cluster converges on a single active node even during network partitions or simultaneous startups.

Is root privilege required to run the Floating-IP plugin?

Yes. The plugin requires root privileges to manage network interfaces. Specifically, the SwitchToActiveMode function invokes net.SetupAlias to create IP aliases on the host interface, and the ARP maintenance goroutine uses raw socket operations to send gratuitous ARP packets. The Setup() function in floating_ip.go validates root access during initialization to prevent runtime permission errors during failover operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}