# INFINI Gateway High Availability and Automatic Failover: Floating-IP Plugin Architecture

> Explore INFINI Gateway's high availability and automatic failover using its Floating-IP plugin. Discover TCP heartbeat, UDP multicast, and leader election for seamless node coordination.

- Repository: [INFINI Labs/gateway](https://github.com/infinilabs/gateway)
- Tags: architecture
- Published: 2026-03-04

---

**INFINI Gateway achieves high availability and automatic failover through a Floating-IP plugin that uses TCP heartbeat monitoring, UDP multicast discovery, and priority-based leader election to coordinate seamless failover between active and standby nodes.**

The `infinilabs/gateway` repository implements a self-contained high availability solution that requires no external coordination services like ZooKeeper or etcd. By leveraging the **Floating-IP plugin**, the gateway provides **INFINI Gateway high availability and automatic failover** capabilities through a lightweight state machine built on standard Linux networking primitives and Go's concurrency patterns.

## Floating-IP Plugin Architecture

The high availability system centers on a state machine that manages two distinct roles: **active** (holds the floating IP) and **standby** (monitors the active node). This logic is implemented primarily in [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go).

### State Machine and Initial Role Detection

Upon startup, the `StateMachine()` function (lines 61-84) determines whether the local node should start as active or standby by probing the configured floating IP address:

```go
client := heartbeat.New()
aliveChan := make(chan bool)
go func() {
    err := client.Start(floatingIPConfig.IP, floatingIPConfig.Echo.EchoPort,
        floatingIPConfig.Echo.EchoDialTimeout, floatingIPConfig.Echo.EchoTimeout,
        func() { aliveChan <- true },   // on success
        func() { aliveChan <- false }) // on failure
    if err != nil { aliveChan <- false }
}()
alive := <-aliveChan

```

If the heartbeat client successfully connects to an existing echo service on the floating IP, the node assumes **standby** mode via `SwitchToStandbyMode`. If the probe fails (indicating no active node is present), the node calls `SwitchToActiveMode` to assume leadership.

### Active Mode Implementation

When a node transitions to active via `SwitchToActiveMode` (lines 82-129), it executes three critical operations:

1. **Atomic state update**: Sets the `atomicActiveOrNot` atomic value to `true` to signal goroutines to begin maintenance tasks.
2. **Network alias configuration**: Assigns the floating IP to the local network interface using `net.SetupAlias`:
   ```go
   err := net.SetupAlias(floatingIPConfig.Interface,
                         floatingIPConfig.IP, floatingIPConfig.Netmask)
   ```

3. **Background maintenance goroutines**:
   - **Gratuitous ARP**: Every 10 seconds, `arping.GratuitousArpOverIfaceByName` broadcasts ARP updates to ensure L2 switches update their MAC tables.
   - **Multicast status broadcast**: Every 10 seconds, `Broadcast(&floatingIPConfig, &req)` sends UDP packets containing the node's active status, priority, and echo port to the multicast group.

Both goroutines monitor `atomicActiveOrNot` and terminate automatically if the node is demoted to standby.

### Standby Mode and Automatic Failover

When a node enters `SwitchToStandbyMode` (lines 80-119), it:

1. Clears the atomic flag to `false`.
2. Invokes `Deactivate(false)` to remove the network alias and stop ARP/broadcast goroutines.
3. Launches a **watchdog goroutine** that continuously monitors the active node:

```go
client.Start(floatingIPConfig.IP, floatingIPConfig.Echo.EchoPort,
            floatingIPConfig.Echo.EchoDialTimeout, floatingIPConfig.Echo.EchoTimeout,
            func() { aliveChan <- true },
            func() { aliveChan <- false })

```

If the heartbeat client reports a failure (connection timeout or lost acknowledgement), the standby node immediately invokes `module.SwitchToActiveMode()`, seizing the floating IP and broadcasting its new status. This completes the **automatic failover** within seconds of detecting a failure.

## Heartbeat Protocol and Health Monitoring

The health checking system operates through a custom TCP protocol implemented in [`service/heartbeat/server.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/server.go) and [`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go).

### TCP Echo Service

The heartbeat server listens on a configurable port (default 61111) and implements a lightweight binary protocol:

| Direction | Byte Value | Meaning |
|-----------|------------|---------|
| Client → Server | `0x01` (Req_REGISTER) | Client registration |
| Server → Client | `0x02` (Res_REGISTER) | Registration ACK |
| Server → Client | `0x03` (Req_HEARTBEAT) | Heartbeat request |
| Client → Server | `0x04` (Res_HEARTBEAT) | Heartbeat response |

### Heartbeat Client and Failover Triggers

The client ([`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go)) maintains a persistent connection to the active node's echo port. It periodically sends `Req_HEARTBEAT` messages and expects `Res_HEARTBEAT` responses within the configured timeout. 

When the client detects a timeout or connection reset, it invokes the failure callback registered by the floating_ip module. This callback triggers the standby node's promotion logic, ensuring **automatic failover** occurs without manual intervention.

## Multicast Discovery and Priority-Based Election

To prevent split-brain scenarios and coordinate multiple standby nodes, the system uses UDP multicast discovery defined in [`service/floating_ip/broadcast.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/broadcast.go).

### UDP Broadcast Protocol

Nodes broadcast their status via the `Request` struct (lines 53-58):

```go
type Request struct {
    IsActive   bool
    FloatingIP string
    FixedIP    string
    EchoPort   int
    Priority   int
}

```

Each node transmits these packets to a configurable multicast address (default `224.3.2.2:7654`) every 10 seconds. The `Priority` field enables deterministic leader election when multiple nodes compete for the active role.

### Split-Brain Avoidance

The multicast handler in [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go) (`ServeMulticastDiscovery`, lines 97-138) implements conflict resolution:

- **Self-detection**: Ignores packets where `FixedIP` matches the local address.
- **Priority-based demotion**: If an active node receives a broadcast from a higher-priority node, it voluntarily demotes itself to standby via `SwitchToStandbyMode`.
- **Preemptive takeover**: When `ForcedSwitchByPriority` is enabled, a standby node that detects a lower-priority active node can preemptively seize leadership and rebroadcast its status.

This mechanism ensures that even during network partitions or simultaneous node startups, the cluster converges on a single active node with the highest configured priority.

## Key Implementation Files

| File | Responsibility |
|------|----------------|
| [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go) | Core state machine, role transitions, ARP management, and multicast handling |
| [`service/floating_ip/broadcast.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/broadcast.go) | UDP multicast packet construction and transmission |
| [`service/heartbeat/server.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/server.go) | TCP echo service for health checks |
| [`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go) | Heartbeat client with failure callbacks |
| [`gateway.yml`](https://github.com/infinilabs/gateway/blob/main/gateway.yml) | Configuration for floating IP, netmask, interface, priority, and multicast settings |

## Practical Configuration Examples

### Starting the Floating-IP Plugin

```go
// In your main.go or plugin loader
import "infini.sh/gateway/service/floating_ip"

func main() {
    plugin := floating_ip.FloatingIPPlugin{}
    plugin.Setup()   // read config, verify root
    if err := plugin.Start(); err != nil {
        log.Fatalf("failed to start floating_ip: %v", err)
    }
    // Run other gateway components ...
}

```

*Relevant source:* [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go) – `Setup`, `Start`.

### Manually Triggering Role Changes

```go
// Force this node to become active
plugin.SwitchToActiveMode()

// Later, demote it back to standby after a short delay
time.Sleep(10 * time.Second)
plugin.SwitchToStandbyMode(0)

```

*Relevant source:* `SwitchToActiveMode` and `SwitchToStandbyMode` in [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go).

### Using the Heartbeat Client Directly

```go
hb := heartbeat.New()
hb.Start(
    "10.0.0.5",                 // target IP (active node)
    61111,                      // echo port
    5* time.Second,             // dial timeout
    10* time.Second,            // read timeout
    func() { log.Println("active node alive") },
    func() { log.Println("active node down – promote!") })

```

*Relevant source:* [`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go).

### Sending Custom Multicast Broadcasts

```go
cfg := &floating_ip.FloatingIPConfig{
    BroadcastConfig: config.NetworkConfig{Binding: "224.3.2.2:7654"},
    // other fields omitted for brevity
}
req := &floating_ip.Request{
    IsActive:   true,
    FloatingIP: "192.168.1.100",
    FixedIP:    "192.168.1.10",
    EchoPort:   61111,
    Priority:   900,
}
floating_ip.Broadcast(cfg, req)

```

*Relevant source:* [`service/floating_ip/broadcast.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/broadcast.go).

## Summary

- **INFINI Gateway high availability and automatic failover** are implemented through a self-contained Floating-IP plugin that requires no external coordination services.
- The **state machine** in [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go) determines initial roles by probing the heartbeat echo service on the floating IP address.
- **Active nodes** assume the floating IP using Linux network aliases and gratuitous ARP, while broadcasting their status via UDP multicast every 10 seconds.
- **Standby nodes** continuously monitor the active node through the heartbeat client in [`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go), automatically promoting themselves upon detecting a failure.
- **Priority-based election** and multicast discovery prevent split-brain scenarios, ensuring the highest-priority healthy node always maintains the active role.

## Frequently Asked Questions

### How does INFINI Gateway detect when the active node has failed?

The standby node runs a heartbeat client ([`service/heartbeat/client.go`](https://github.com/infinilabs/gateway/blob/main/service/heartbeat/client.go)) that periodically connects to the active node's TCP echo port (default 61111). If the connection times out or the heartbeat acknowledgement is not received within the configured interval, the client invokes a failure callback. This callback triggers `SwitchToActiveMode()` in the floating_ip module, causing the standby node to seize the floating IP and assume the active role.

### Can multiple standby nodes exist in the same cluster?

Yes. Any number of nodes can operate in standby mode simultaneously. Each standby independently monitors the active node via heartbeats and listens to multicast broadcasts. If the active node fails, all standbys will attempt to promote themselves. However, the priority-based election mechanism ensures that only the node with the highest configured priority successfully transitions to active mode, while others remain in or return to standby.

### What prevents two nodes from both claiming the active role simultaneously?

The system implements **split-brain avoidance** through the multicast discovery handler in [`service/floating_ip/floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/service/floating_ip/floating_ip.go). When a node receives a broadcast from another node claiming to be active, it compares priorities. An active node that discovers a higher-priority peer will voluntarily demote itself to standby. Additionally, if `ForcedSwitchByPriority` is enabled, a standby node can preempt a lower-priority active node. These rules ensure the cluster converges on a single active node even during network partitions or simultaneous startups.

### Is root privilege required to run the Floating-IP plugin?

Yes. The plugin requires root privileges to manage network interfaces. Specifically, the `SwitchToActiveMode` function invokes `net.SetupAlias` to create IP aliases on the host interface, and the ARP maintenance goroutine uses raw socket operations to send gratuitous ARP packets. The `Setup()` function in [`floating_ip.go`](https://github.com/infinilabs/gateway/blob/main/floating_ip.go) validates root access during initialization to prevent runtime permission errors during failover operations.