How to Configure High Availability and Clustering for INFINI Console: 3-Tier Resilience Strategy

Run multiple INFINI Console instances behind a load balancer with shared Badger storage, configure comma-separated endpoints for your ingest cluster, and register monitored clusters with multiple host entries to achieve full-stack high availability.

INFINI Console is an open-source management platform for Elasticsearch clusters that requires careful configuration to ensure resilience in production environments. To configure high availability and clustering for INFINI Console effectively, you must address three distinct layers: the Console service itself, the ingest cluster storing metrics, and the monitored Elasticsearch clusters. This guide walks through each layer using the actual source configuration files from the infinilabs/console repository.

Console-Level High Availability for INFINI Console

Console-level HA ensures the INFINI Console UI and API remain available even if individual server instances fail. This requires running multiple Console binaries behind a reverse proxy and ensuring all instances share the same metadata store.

Deploying Multiple Instances Behind a Load Balancer

Deploy the infini-console binary to at least three hosts. In console.yml, bind the web server to all interfaces so the load balancer can reach any instance:

# console.yml

web:
  enabled: true
  network:
    binding: 0.0.0.0:9000
    skip_occupied_port: true

Source: console.yml lines 36-38 (view source)

Place an HAProxy, Nginx, or Kubernetes Service in front of these instances to distribute traffic. Session affinity is not required because state is stored in the shared Badger database.

Configuring Shared Badger Storage

By default, each Console instance maintains local state in a Badger KV store. For HA, point all instances to the same network-mounted directory:

# console.yml

badger:
  enabled: true
  path: "/shared/badger-data"  # NFS, EFS, or SAN mount

Source: console.yml lines 78-84 (view source)

This ensures that metadata, audit logs, and UI state remain consistent across all Console nodes. If you cannot use shared storage, consider configuring an external KV store such as etcd.

Enabling Configuration Auto-Reload

To apply changes to cluster definitions without restarting every instance, enable auto-reload:

# console.yml

configs:
  managed: false
  auto_reload: true

Source: console.yml lines 15-17 (view source)

When you update system_config.tpl or other configuration files, all Console instances will pick up the changes within seconds.

Ingest Cluster High Availability Configuration

The Console stores metrics, logs, and audit data in a dedicated Elasticsearch cluster called the ingest cluster. Configuring high availability for this backend prevents data loss and ensures the Console can continue collecting telemetry even if individual Elasticsearch nodes fail.

Multi-Node Endpoint Configuration

In config/system_config.tpl, define the ingest cluster with a comma-separated list of nodes:

# config/system_config.tpl

elasticsearch:
  - id: system-cluster
    name: system-cluster
    enabled: true
    monitored: true
    reserved: true
    endpoint: http://es-node1:9200,http://es-node2:9200,http://es-node3:9200
    basic_auth:
      username: $[[CLUSTER_USER]]
      password: $[[keystore.SYSTEM_CLUSTER_PASS]]

The internal Elasticsearch client automatically round-robins requests across these endpoints. If one node returns a connection error, the client retries the request on the next available node without raising an error to the Console application.

Health Check and Failover Settings

Enable periodic health checks so the Console can mark the ingest cluster as unavailable only when all hosts are down:

# console.yml

elastic:
  health_check:
    enabled: true
    interval: 30s
  availability_check:
    enabled: true
    interval: 60s

Source: console.yml lines 45-51 (view source)

The health_check task pings the ingest cluster every 30 seconds. The availability_check task (used for monitored clusters) runs every 60 seconds. Together they ensure that the Console accurately reflects cluster state while tolerating transient network issues.

Monitored Cluster High Availability

When you register external Elasticsearch clusters for monitoring, you can provide multiple host entries so the Console can fail over if a node becomes unreachable.

Multi-Host Registration (v1.28.1+)

Support for multiple hosts was introduced in Console v1.28.1. When creating a cluster via the REST API or UI, supply a hosts array:

POST /api/v1/cluster
{
  "id": "prod-es",
  "name": "Production ES",
  "hosts": [
    "https://es-node1:9200",
    "https://es-node2:9200",
    "https://es-node3:9200"
  ],
  "basic_auth": {
    "username": "admin",
    "password": "******"
  },
  "enabled": true,
  "monitored": true
}

Source: Release notes confirming v1.28.1 feature (view source)

The validation logic in plugin/setup/setup.go parses this list and ensures at least one host is reachable before marking the cluster as valid.

Availability Check Logic

The Console runs a node_availability_check task that verifies at least one host in the list responds to HTTP requests. If a node fails, the Console marks it as unavailable but continues collecting metrics from the remaining hosts.

Pipeline definitions in config/system_config.tpl use the cluster_available condition to pause indexing when no hosts are reachable:

# config/system_config.tpl

pipeline:
  - name: merge_metrics
    auto_start: true
    keep_running: true
    processor:
      - indexing_merge:
          input_queue: "metrics"
          elasticsearch: "$[[CLUSTER_ID]]"
          index_name: "$[[INDEX_PREFIX]]metrics"
          when:
            cluster_available: ["$[[CLUSTER_ID]]"]

Source: config/system_config.tpl lines 46-48 (view source)

This prevents the Console from attempting to write metrics to a completely unavailable cluster, reducing error noise and retry overhead.

End-to-End HA Deployment Example

Combine all three layers into a production-ready deployment. Below is a minimal architecture using three Console instances, a shared Badger store, and a three-node Elasticsearch ingest cluster.

Architecture Overview

• Console Tier: Three nodes (console-01, console-02, console-03) behind an HAProxy load balancer.
• Storage Tier: NFS mount at /shared/badger-data accessible by all Console nodes.
• Ingest Tier: Three-node Elasticsearch cluster (es-node1, es-node2, es-node3) storing Console metrics.

Step 1: Prepare Shared Storage

Mount an NFS share on all Console hosts:

# On console-01, console-02, console-03

sudo mount -t nfs nfs-server:/exports/badger /shared/badger-data

Step 2: Deploy Console Configuration

Install the following console.yml on all three nodes:

path.configs: "config"
configs:
  managed: false
  auto_reload: true

web:
  enabled: true
  network:
    binding: 0.0.0.0:9000
    skip_occupied_port: true

elastic:
  enabled: true
  remote_configs: true
  health_check:
    enabled: true
    interval: 30s
  availability_check:
    enabled: true
    interval: 60s
  orm:
    enabled: true
    index_prefix: ".infini_"

badger:
  enabled: true
  path: "/shared/badger-data"

security:
  enabled: true

Step 3: Configure the Ingest Cluster

Create config/system_config.tpl with multiple endpoints:

elasticsearch:
  - id: system-cluster
    name: system-cluster
    enabled: true
    monitored: true
    reserved: true
    endpoint: http://es-node1:9200,http://es-node2:9200,http://es-node3:9200
    basic_auth:
      username: elastic
      password: $[[keystore.SYSTEM_CLUSTER_PASS]]

pipeline:
  - name: merge_metrics
    auto_start: true
    keep_running: true
    processor:
      - indexing_merge:
          input_queue: "metrics"
          elasticsearch: "system-cluster"
          index_name: ".infini_metrics"
          when:
            cluster_available: ["system-cluster"]

Step 4: Start Services

Start Console on all nodes:

./infini-console -config console.yml

Configure HAProxy to balance traffic across the three instances:

frontend console_frontend
    bind *:80
    default_backend console_backend

backend console_backend
    balance roundrobin
    server console-01 10.0.1.10:9000 check
    server console-02 10.0.1.11:9000 check
    server console-03 10.0.1.12:9000 check

Step 5: Register Monitored Clusters with Multiple Hosts

When adding production clusters via the API, include all node addresses:

curl -X POST http://console-lb/api/v1/cluster \
  -H "Content-Type: application/json" \
  -d '{
    "id": "prod-cluster",
    "name": "Production Cluster",
    "hosts": [
      "https://prod-es-01:9200",
      "https://prod-es-02:9200",
      "https://prod-es-03:9200"
    ],
    "basic_auth": {
      "username": "admin",
      "password": "secure-password"
    },
    "enabled": true,
    "monitored": true
  }'

Troubleshooting High Availability Issues

Symptom	Likely Cause	Fix
Console UI shows System cluster unavailable	All ingest-cluster hosts are down or network partitioned.	Verify the endpoint list in system_config.tpl includes reachable nodes and ensure the shared Badger store is accessible.
Monitored cluster shows degraded after node crash	The node_availability_check task flagged the host as down.	Confirm remaining host URLs are correct; Console automatically continues using reachable nodes.
Configuration changes not applied	configs.auto_reload is disabled.	Set auto_reload: true in console.yml and verify file permissions allow the Console process to read the config directory.
Duplicate metadata after re-registration	Badger KV is not shared across Console nodes.	Mount a shared filesystem (NFS, EFS) at the badger.path location or migrate to an external KV store.

Summary

To configure high availability and clustering for INFINI Console, implement resilience at three distinct layers:

• Console Service HA: Deploy multiple infini-console binaries behind a load balancer, enable configs.auto_reload, and share a single Badger KV directory across all instances to maintain consistent metadata.
• Ingest Cluster HA: Define multiple Elasticsearch nodes in config/system_config.tpl using comma-separated endpoints, enable elastic.health_check, and use the cluster_available condition in pipelines to prevent writes during total outages.
• Monitored Cluster HA: Register external clusters with a hosts array (available since v1.28.1) so the node_availability_check task can fail over to healthy nodes automatically.

Frequently Asked Questions

How many Console instances should I run for production high availability?

Run at least three Console instances behind a load balancer. This ensures that during rolling updates or single-node failures, a quorum of instances remains available to serve the UI and API. According to the source configuration in console.yml, all instances must share the same badger.path to prevent metadata drift.

Can I use a database instead of Badger for shared storage?

Yes, while Badger is the default embedded KV store, you can configure an external backend. Set badger.enabled: false and configure the appropriate ORM settings in console.yml to point to a shared database. However, the most common production pattern shown in console.yml (lines 78-84) uses a network-mounted Badger directory for simplicity and performance.

What happens if all ingest cluster nodes become unreachable?

If every host in the ingest cluster endpoint list fails health checks, the Console will mark the system cluster as unavailable and pause metric collection pipelines. The cluster_available condition in config/system_config.tpl (lines 46-48) prevents the Console from attempting to write to a completely dead cluster, reducing error noise. The UI will display a warning, but the Console service itself remains operational and will resume ingestion automatically when at least one node recovers.

How do I verify that multi-host failovers are working correctly?

Test failover by deliberately stopping one Elasticsearch node while monitoring the Console logs. For the ingest cluster, you should see the health check continue passing as long as other nodes respond. For monitored clusters, check the node_availability_check logs (enabled by default with a 60-second interval in console.yml lines 48-51). The Console UI should show the cluster status as "degraded" rather than "offline" when some hosts are unreachable, confirming that failover logic is active.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}