How to Ensure Zero-Downtime During Elasticsearch Cluster Topology Changes with INFINI Gateway

INFINI Gateway achieves zero-downtime during Elasticsearch cluster topology changes by dynamically discovering nodes, periodically refreshing upstream pools based on topology version changes, and gracefully switching balancers while maintaining existing connections.

The INFINI Gateway proxy (from the infinilabs/gateway repository) sits between your applications and Elasticsearch clusters, insulating clients from backend instability. When nodes join, leave, or fail within the Elasticsearch cluster, the gateway must adapt without dropping active requests or forcing client reconnections. The following mechanisms work together to provide continuous availability during these topology transitions.

Dynamic Node Discovery and Topology Version Tracking

The gateway monitors the Elasticsearch metadata service to detect cluster changes immediately. In proxy/output/elastic/reverseproxy.go, the refreshNodes method compares the current NodesTopologyVersion against a cached version to determine if the upstream pool needs rebuilding:

if oldV == p.lastNodesTopologyVersion {
    return
}

This check appears at lines 81–89 in reverseproxy.go. Only when the version differs does the gateway proceed to reconstruct the host list, ensuring that computational overhead is minimized during stable periods while reacting instantly to actual topology changes.

Enable this behavior in your configuration by setting discovery.enabled: true within your Elasticsearch filter configuration. The gateway then subscribes to metadata updates and automatically incorporates new nodes into the routing pool as soon as they advertise their HTTP endpoints.

Periodic Refresh Task Configuration

To guarantee that version changes are detected even if the initial metadata push fails, the gateway registers a scheduled background task during proxy initialization. In reverseproxy.go at lines 25–35, the NewReverseProxy function registers the refresh task when cfg.Refresh.Enabled evaluates to true:

task2.RegisterScheduleTask(task)

This task repeatedly invokes refreshNodes(false) at the interval specified in your configuration. For production environments, set a short but reasonable interval to balance responsiveness against overhead:

refresh:
  enabled: true
  interval: "5s"

A five-second interval ensures that new nodes begin receiving traffic within moments of joining the cluster, while removed nodes are purged from the rotation before subsequent requests attempt to reach them.

Graceful Balancer Reconstruction

When the topology version changes, the gateway rebuilds its load balancer without severing existing connections. After computing the newHosts slice in reverseproxy.go (lines 96–100), the code atomically swaps the endpoint slice and instantiates a fresh balancer:

p.endpoints = newHosts
p.bla = balancer.NewBalancer(ws)

Existing connections to the previous node list continue until they naturally close, while new requests immediately route through the updated balancer. This design prevents abrupt connection termination for in-flight requests while ensuring that fresh traffic avoids nodes that have left the cluster.

Health-Check Fallback Protection

Even with dynamic discovery, individual nodes may become unresponsive before the next topology refresh occurs. The gateway implements defensive health checking in proxy/output/elastic/elasticsearch.go (lines 65–74). Before returning a 503 Service Unavailable response, the filter verifies cluster state:

if !metadata.IsAvailable() {
    // Optional: perform explicit health check
    elastic.GetClient(...).ClusterHealth(nil)
}

Configure skip_available_check: false (the default) to enforce availability validation on every request. Additionally, set check_cluster_health_when_not_available: true to ensure the gateway performs an explicit ClusterHealth call before declaring the backend unavailable, preventing false positives during momentary network jitter.

Recommended Configuration for Production

Combine these mechanisms in your gateway.yml or dedicated filter configuration file to achieve zero-downtime operation:

filters:
  elasticsearch:
    elasticsearch: "myCluster"
    balancer: "weight"
    max_connection_per_node: 5000
    max_retry_times: 5
    retry_on_backend_failure: true
    discovery:
      enabled: true
    refresh:
      enabled: true
      interval: "5s"
    skip_available_check: false
    check_cluster_health_when_not_available: true
    weights:
      "10.0.0.1:9200": 2
      "10.0.0.2:9200": 1

The refresh.interval parameter controls how quickly the gateway reacts to topology changes. When a new node joins and the metadata service increments NodesTopologyVersion, the next refresh iteration rebuilds the host list and updates the balancer instantly.

Manual Operations and Testing

Triggering Manual Refreshes

For testing or emergency scenarios, you can force a topology refresh programmatically using the same internal method invoked by the scheduled task. In reverseproxy.go at lines 55–58, the refreshNodes function accepts a boolean force parameter:

import "infini.sh/gateway/proxy/output/elastic"

func forceRefresh(rp *elastic.ReverseProxy) {
    rp.refreshNodes(true) // true forces refresh even if discovery is disabled
}

Adding Nodes Without Restart

Once a new Elasticsearch node joins the cluster and advertises its HTTP transport address, no gateway restart or code changes are required. The metadata service automatically updates the topology version, and the scheduled refresh task picks up the change within the configured interval.

Customizing Health-Check Behavior

Adjust the rate limiting on cluster health checks to match your SLA requirements. The default implementation in elasticsearch.go (lines 66–73) uses a rate limiter to prevent health check storms:

rate.GetRateLimiter("cluster_check_health", metadata.Config.ID, 1, 1, 30*time.Second).Allow()

Modify the limiter parameters (burst size and duration) to increase or decrease health check frequency during instability periods.

Summary

• Dynamic discovery tracks NodesTopologyVersion in reverseproxy.go to detect cluster changes without polling overhead.
• Periodic refresh tasks ensure topology updates are captured even if metadata pushes fail, configurable via refresh.interval.
• Graceful balancer switches rebuild the routing table atomically while allowing existing connections to drain naturally.
• Health-check fallbacks validate node availability before returning 503 errors, configurable through skip_available_check and check_cluster_health_when_not_available flags.
• Fixed-client mode should be avoided (fixed_client: false) for zero-downtime guarantees, as it bypasses the dynamic refresh logic.

Frequently Asked Questions

How does INFINI Gateway detect when Elasticsearch nodes are added or removed?

The gateway monitors metadata.NodesTopologyVersion via the refreshNodes method in proxy/output/elastic/reverseproxy.go. When the metadata service reports a version increment—indicating a node join, leave, or failure—the cached version comparison fails, triggering a rebuild of the upstream host list and balancer.

What happens to in-flight requests when a node leaves the cluster?

Existing connections to departed nodes continue processing until they naturally close, as the endpoints slice and balancer swap occur atomically without connection termination. New requests immediately route to the updated healthy node list, ensuring no abrupt disconnections for active operations.

Can I use a fixed client configuration and still achieve zero-downtime?

No. Fixed-client mode (fixed_client: true), implemented in reverseproxy.go at lines 16–22, binds the proxy to a single static host. This disables the dynamic refresh logic and topology version tracking, eliminating the zero-downtime guarantees during cluster changes. Use dynamic mode for production clusters.

How often should I set the refresh interval for production clusters?

Set refresh.interval to a value between 5 and 10 seconds for most production workloads. This interval, defined in proxy/output/elastic/config.go, ensures rapid detection of topology changes without overwhelming the gateway or Elasticsearch metadata service with excessive polling operations.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}