# How to Configure High Availability and Clustering for INFINI Console: 3-Tier Resilience Strategy

> Achieve INFINI Console high availability and clustering with a 3-tier resilience strategy. Learn to configure load balancing, shared storage, and multiple host entries for full-stack resilience.

- Repository: [INFINI Labs/console](https://github.com/infinilabs/console)
- Tags: how-to-guide
- Published: 2026-03-04

---

**Run multiple INFINI Console instances behind a load balancer with shared Badger storage, configure comma-separated endpoints for your ingest cluster, and register monitored clusters with multiple host entries to achieve full-stack high availability.**

INFINI Console is an open-source management platform for Elasticsearch clusters that requires careful configuration to ensure resilience in production environments. To configure high availability and clustering for INFINI Console effectively, you must address three distinct layers: the Console service itself, the ingest cluster storing metrics, and the monitored Elasticsearch clusters. This guide walks through each layer using the actual source configuration files from the `infinilabs/console` repository.

## Console-Level High Availability for INFINI Console

Console-level HA ensures the INFINI Console UI and API remain available even if individual server instances fail. This requires running multiple Console binaries behind a reverse proxy and ensuring all instances share the same metadata store.

### Deploying Multiple Instances Behind a Load Balancer

Deploy the `infini-console` binary to at least three hosts. In [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml), bind the web server to all interfaces so the load balancer can reach any instance:

```yaml

# console.yml

web:
  enabled: true
  network:
    binding: 0.0.0.0:9000
    skip_occupied_port: true

```

Source: [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) lines 36-38 ([view source](https://github.com/infinilabs/console/blob/main/console.yml#L36-L38))

Place an HAProxy, Nginx, or Kubernetes Service in front of these instances to distribute traffic. Session affinity is not required because state is stored in the shared Badger database.

### Configuring Shared Badger Storage

By default, each Console instance maintains local state in a Badger KV store. For HA, point all instances to the same network-mounted directory:

```yaml

# console.yml

badger:
  enabled: true
  path: "/shared/badger-data"  # NFS, EFS, or SAN mount

```

Source: [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) lines 78-84 ([view source](https://github.com/infinilabs/console/blob/main/console.yml#L78-L84))

This ensures that metadata, audit logs, and UI state remain consistent across all Console nodes. If you cannot use shared storage, consider configuring an external KV store such as etcd.

### Enabling Configuration Auto-Reload

To apply changes to cluster definitions without restarting every instance, enable auto-reload:

```yaml

# console.yml

configs:
  managed: false
  auto_reload: true

```

Source: [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) lines 15-17 ([view source](https://github.com/infinilabs/console/blob/main/console.yml#L15-L17))

When you update `system_config.tpl` or other configuration files, all Console instances will pick up the changes within seconds.

## Ingest Cluster High Availability Configuration

The Console stores metrics, logs, and audit data in a dedicated Elasticsearch cluster called the **ingest cluster**. Configuring high availability for this backend prevents data loss and ensures the Console can continue collecting telemetry even if individual Elasticsearch nodes fail.

### Multi-Node Endpoint Configuration

In `config/system_config.tpl`, define the ingest cluster with a comma-separated list of nodes:

```yaml

# config/system_config.tpl

elasticsearch:
  - id: system-cluster
    name: system-cluster
    enabled: true
    monitored: true
    reserved: true
    endpoint: http://es-node1:9200,http://es-node2:9200,http://es-node3:9200
    basic_auth:
      username: $[[CLUSTER_USER]]
      password: $[[keystore.SYSTEM_CLUSTER_PASS]]

```

The internal Elasticsearch client automatically round-robins requests across these endpoints. If one node returns a connection error, the client retries the request on the next available node without raising an error to the Console application.

### Health Check and Failover Settings

Enable periodic health checks so the Console can mark the ingest cluster as unavailable only when all hosts are down:

```yaml

# console.yml

elastic:
  health_check:
    enabled: true
    interval: 30s
  availability_check:
    enabled: true
    interval: 60s

```

Source: [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) lines 45-51 ([view source](https://github.com/infinilabs/console/blob/main/console.yml#L45-L51))

The `health_check` task pings the ingest cluster every 30 seconds. The `availability_check` task (used for monitored clusters) runs every 60 seconds. Together they ensure that the Console accurately reflects cluster state while tolerating transient network issues.

## Monitored Cluster High Availability

When you register external Elasticsearch clusters for monitoring, you can provide multiple host entries so the Console can fail over if a node becomes unreachable.

### Multi-Host Registration (v1.28.1+)

Support for multiple hosts was introduced in Console v1.28.1. When creating a cluster via the REST API or UI, supply a `hosts` array:

```json
POST /api/v1/cluster
{
  "id": "prod-es",
  "name": "Production ES",
  "hosts": [
    "https://es-node1:9200",
    "https://es-node2:9200",
    "https://es-node3:9200"
  ],
  "basic_auth": {
    "username": "admin",
    "password": "******"
  },
  "enabled": true,
  "monitored": true
}

```

Source: Release notes confirming v1.28.1 feature ([view source](https://github.com/infinilabs/console/blob/main/docs/content.en/docs/release-notes/_index.md#L72-L74))

The validation logic in [`plugin/setup/setup.go`](https://github.com/infinilabs/console/blob/main/plugin/setup/setup.go) parses this list and ensures at least one host is reachable before marking the cluster as valid.

### Availability Check Logic

The Console runs a `node_availability_check` task that verifies at least one host in the list responds to HTTP requests. If a node fails, the Console marks it as unavailable but continues collecting metrics from the remaining hosts.

Pipeline definitions in `config/system_config.tpl` use the `cluster_available` condition to pause indexing when no hosts are reachable:

```yaml

# config/system_config.tpl

pipeline:
  - name: merge_metrics
    auto_start: true
    keep_running: true
    processor:
      - indexing_merge:
          input_queue: "metrics"
          elasticsearch: "$[[CLUSTER_ID]]"
          index_name: "$[[INDEX_PREFIX]]metrics"
          when:
            cluster_available: ["$[[CLUSTER_ID]]"]

```

Source: `config/system_config.tpl` lines 46-48 ([view source](https://github.com/infinilabs/console/blob/main/config/system_config.tpl#L46-L48))

This prevents the Console from attempting to write metrics to a completely unavailable cluster, reducing error noise and retry overhead.

## End-to-End HA Deployment Example

Combine all three layers into a production-ready deployment. Below is a minimal architecture using three Console instances, a shared Badger store, and a three-node Elasticsearch ingest cluster.

**Architecture Overview**

- **Console Tier**: Three nodes (`console-01`, `console-02`, `console-03`) behind an HAProxy load balancer.
- **Storage Tier**: NFS mount at `/shared/badger-data` accessible by all Console nodes.
- **Ingest Tier**: Three-node Elasticsearch cluster (`es-node1`, `es-node2`, `es-node3`) storing Console metrics.

**Step 1: Prepare Shared Storage**

Mount an NFS share on all Console hosts:

```bash

# On console-01, console-02, console-03

sudo mount -t nfs nfs-server:/exports/badger /shared/badger-data

```

**Step 2: Deploy Console Configuration**

Install the following [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) on all three nodes:

```yaml
path.configs: "config"
configs:
  managed: false
  auto_reload: true

web:
  enabled: true
  network:
    binding: 0.0.0.0:9000
    skip_occupied_port: true

elastic:
  enabled: true
  remote_configs: true
  health_check:
    enabled: true
    interval: 30s
  availability_check:
    enabled: true
    interval: 60s
  orm:
    enabled: true
    index_prefix: ".infini_"

badger:
  enabled: true
  path: "/shared/badger-data"

security:
  enabled: true

```

**Step 3: Configure the Ingest Cluster**

Create `config/system_config.tpl` with multiple endpoints:

```yaml
elasticsearch:
  - id: system-cluster
    name: system-cluster
    enabled: true
    monitored: true
    reserved: true
    endpoint: http://es-node1:9200,http://es-node2:9200,http://es-node3:9200
    basic_auth:
      username: elastic
      password: $[[keystore.SYSTEM_CLUSTER_PASS]]

pipeline:
  - name: merge_metrics
    auto_start: true
    keep_running: true
    processor:
      - indexing_merge:
          input_queue: "metrics"
          elasticsearch: "system-cluster"
          index_name: ".infini_metrics"
          when:
            cluster_available: ["system-cluster"]

```

**Step 4: Start Services**

Start Console on all nodes:

```bash
./infini-console -config console.yml

```

Configure HAProxy to balance traffic across the three instances:

```haproxy
frontend console_frontend
    bind *:80
    default_backend console_backend

backend console_backend
    balance roundrobin
    server console-01 10.0.1.10:9000 check
    server console-02 10.0.1.11:9000 check
    server console-03 10.0.1.12:9000 check

```

**Step 5: Register Monitored Clusters with Multiple Hosts**

When adding production clusters via the API, include all node addresses:

```bash
curl -X POST http://console-lb/api/v1/cluster \
  -H "Content-Type: application/json" \
  -d '{
    "id": "prod-cluster",
    "name": "Production Cluster",
    "hosts": [
      "https://prod-es-01:9200",
      "https://prod-es-02:9200",
      "https://prod-es-03:9200"
    ],
    "basic_auth": {
      "username": "admin",
      "password": "secure-password"
    },
    "enabled": true,
    "monitored": true
  }'

```

## Troubleshooting High Availability Issues

| Symptom | Likely Cause | Fix |
|---------|--------------|-----|
| Console UI shows **System cluster unavailable** | All ingest-cluster hosts are down or network partitioned. | Verify the `endpoint` list in `system_config.tpl` includes reachable nodes and ensure the shared Badger store is accessible. |
| Monitored cluster shows **degraded** after node crash | The `node_availability_check` task flagged the host as down. | Confirm remaining host URLs are correct; Console automatically continues using reachable nodes. |
| Configuration changes not applied | `configs.auto_reload` is disabled. | Set `auto_reload: true` in [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) and verify file permissions allow the Console process to read the config directory. |
| Duplicate metadata after re-registration | Badger KV is not shared across Console nodes. | Mount a shared filesystem (NFS, EFS) at the `badger.path` location or migrate to an external KV store. |

## Summary

To configure high availability and clustering for INFINI Console, implement resilience at three distinct layers:

- **Console Service HA**: Deploy multiple `infini-console` binaries behind a load balancer, enable `configs.auto_reload`, and share a single Badger KV directory across all instances to maintain consistent metadata.
- **Ingest Cluster HA**: Define multiple Elasticsearch nodes in `config/system_config.tpl` using comma-separated endpoints, enable `elastic.health_check`, and use the `cluster_available` condition in pipelines to prevent writes during total outages.
- **Monitored Cluster HA**: Register external clusters with a `hosts` array (available since v1.28.1) so the `node_availability_check` task can fail over to healthy nodes automatically.

## Frequently Asked Questions

### How many Console instances should I run for production high availability?

Run at least three Console instances behind a load balancer. This ensures that during rolling updates or single-node failures, a quorum of instances remains available to serve the UI and API. According to the source configuration in [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml), all instances must share the same `badger.path` to prevent metadata drift.

### Can I use a database instead of Badger for shared storage?

Yes, while Badger is the default embedded KV store, you can configure an external backend. Set `badger.enabled: false` and configure the appropriate ORM settings in [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) to point to a shared database. However, the most common production pattern shown in [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) (lines 78-84) uses a network-mounted Badger directory for simplicity and performance.

### What happens if all ingest cluster nodes become unreachable?

If every host in the ingest cluster `endpoint` list fails health checks, the Console will mark the system cluster as unavailable and pause metric collection pipelines. The `cluster_available` condition in `config/system_config.tpl` (lines 46-48) prevents the Console from attempting to write to a completely dead cluster, reducing error noise. The UI will display a warning, but the Console service itself remains operational and will resume ingestion automatically when at least one node recovers.

### How do I verify that multi-host failovers are working correctly?

Test failover by deliberately stopping one Elasticsearch node while monitoring the Console logs. For the ingest cluster, you should see the health check continue passing as long as other nodes respond. For monitored clusters, check the `node_availability_check` logs (enabled by default with a 60-second interval in [`console.yml`](https://github.com/infinilabs/console/blob/main/console.yml) lines 48-51). The Console UI should show the cluster status as "degraded" rather than "offline" when some hosts are unreachable, confirming that failover logic is active.