INFINI Console Multi-Cluster Management Architecture: Handling Cross-Version Elasticsearch Clusters

INFINI Console implements a centralized control-plane architecture that abstracts heterogeneous Elasticsearch clusters into a unified management surface, automatically adapting to version-specific APIs across ES 1.x through 8.x, OpenSearch, and Easysearch distributions.

The infinilabs/console repository provides a production-grade solution for managing multiple Elasticsearch clusters from a single interface. This article examines how INFINI Console's multi-cluster management architecture enables simultaneous administration of diverse Elasticsearch distributions while maintaining strict security isolation and API compatibility.

Three-Layer Control-Plane Architecture

INFINI Console organizes its multi-cluster capabilities into three distinct logical layers that work together to provide a seamless management experience.

API Gateway Layer

The API Gateway exposes RESTful HTTP endpoints for cluster operations. In modules/elastic/api/init.go, the system registers all Elasticsearch-related routes, while core/elastic.go provides low-level helper methods for request processing. This layer enforces RBAC permissions before routing requests to the appropriate backend cluster.

Cluster Service Layer

The Cluster Service maintains cluster metadata in a dedicated system index (.infini_*). During first-run initialization in plugin/setup/setup.go, the system creates a hidden system cluster identified by GlobalSystemElasticsearchID = "infini_default_system_cluster". This service handles registration, health checks, and version discovery through the adapter.ClusterVersion function, which validates that connected clusters meet minimum version requirements (aborting with VersionTooOld for versions below 5.3).

Execution Engine Layer

The Execution Engine routes user requests to target clusters while adapting queries for version-specific APIs. Located in modules/elastic/api/v1/ and core/elastic.go, this layer builds per-cluster filters and selects appropriate request formats based on the discovered distribution and version.

Multi-Cluster Registration and Lifecycle

System Cluster Initialization

On first startup, plugin/setup/setup.go establishes the system cluster that stores Console's own metadata. This bootstrap process validates the Elasticsearch version and distribution before persisting configuration via the ORM layer (orm.Save).

User-Defined Cluster Registration

Administrators add production clusters through POST /elasticsearch/ endpoints. The handler validates endpoints, checks credentials, and stores metadata in the system index. Each cluster entry includes version and distribution information discovered at registration time.

Security Isolation

Access control operates through the GetClusterFilter function in core/elastic.go. This middleware extracts the user's allowed cluster IDs and injects an Elasticsearch-compatible terms filter:

return util.MapStr{
    "terms": util.MapStr{field: clusterIds},
}, false

This ensures users can only query clusters explicitly assigned to their role, with the filter applied at the Elasticsearch query level.

Cross-Version Elasticsearch Handling

INFINI Console does not lock users to a single Elasticsearch version. Instead, it discovers the target cluster's version and distribution at runtime, adapting requests accordingly.

Runtime Version Discovery

The adapter.ClusterVersion function in plugin/setup/setup.go queries the cluster's root endpoint to extract Version.Number and Version.Distribution. This discovery occurs during initial registration and periodic health checks, enabling dynamic adaptation to cluster upgrades.

Distribution-Specific Branching

Console handles three primary distributions: elastic.Elasticsearch, elastic.Easysearch, and elastic.Opensearch. The system branches logic based on ver.Distribution values. For example, index lifecycle management templates select version-specific DSL files such as template_ilm_1_12_1.tpl for Easysearch versions 1.12.1 and above.

API-Level Adaptation

Handlers in modules/elastic/api/v1/cluster_overview.go use the discovered major version to select appropriate document types (_doc versus doc) and index naming conventions. The initializeTemplate function in plugin/setup/setup.go injects version-specific variables into JSON payloads before transmission, ensuring compatibility across ES 1.x through 8.x APIs.

Code Examples

Registering a New Cluster via HTTP API

curl -X POST http://localhost:8080/elasticsearch/ \
  -H "Authorization: Bearer <jwt>" \
  -d '{
        "host": "es-node-01.example.com",
        "schema": "https",
        "username": "admin",
        "password": "s3cr3t",
        "tags": ["production","region-eu"]
      }'

Console stores this metadata in its system index and validates the cluster version using the same adapter.ClusterVersion logic found in plugin/setup/setup.go.

Retrieving Cluster Metadata in Go

import (
    "infini.sh/framework/core/elastic"
    "infini.sh/console/core"
)

// Get the client for a specific cluster ID
client, err := elastic.GetClient("my-cluster-id")
if err != nil {
    log.Fatal(err)
}

// Discover version/distribution
verInfo, _ := adapter.ClusterVersion(elastic.GetMetadata("my-cluster-id"))
fmt.Printf("Cluster runs %s %s\n", verInfo.Version.Distribution, verInfo.Version.Number)

This leverages the same version discovery mechanism used during initial system setup.

Executing Queries with Permission Isolation

func search(req *http.Request, clusterID string, query []byte) ([]byte, error) {
    // Build the per-user cluster filter
    filter, _ := core.GetClusterFilter(req, "metadata.labels.cluster_id")

    // Merge the filter into the query
    var body map[string]interface{}
    json.Unmarshal(query, &body)
    body["query"] = map[string]interface{}{
        "bool": map[string]interface{}{
            "filter": []interface{}{filter},
        },
    }
    payload, _ := json.Marshal(body)

    client, _ := elastic.GetClient(clusterID)
    return client.Search(payload)
}

The core.GetClusterFilter function injects the appropriate terms clause based on the user's allowed clusters, as implemented in core/elastic.go.

Key Implementation Files

The multi-cluster management system is implemented across the following source files:

• modules/elastic/api/init.go – Registers RESTful HTTP endpoints for cluster operations and API gateway routing.

• core/elastic.go – Implements GetClusterFilter for security isolation and provides low-level cluster communication helpers.

• plugin/setup/setup.go – Handles first-run bootstrap, system cluster creation (GlobalSystemElasticsearchID), version validation via adapter.ClusterVersion, and distribution-specific template initialization.

• modules/elastic/api/v1/cluster_overview.go – Implements version-aware cluster information retrieval and metrics collection.

• modules/elastic/api/trace_template.go – Manages version-specific template rendering for index lifecycle management.

• modules/elastic/api/alias.go – Demonstrates proxy patterns for alias operations to target clusters.

• modules/elastic/api/search.go – Handles request routing and search API adaptation across different versions.

Summary

• INFINI Console employs a three-layer control-plane architecture (API Gateway, Cluster Service, Execution Engine) to abstract multiple Elasticsearch clusters into a unified management interface.

• The system maintains cluster metadata in a dedicated system index (.infini_*), with a bootstrap system cluster (infini_default_system_cluster) created during first-run initialization in plugin/setup/setup.go.

• Cross-version compatibility is achieved through runtime version discovery (adapter.ClusterVersion) and distribution-specific branching that adapts API calls for Elasticsearch 1.x-8.x, OpenSearch, and Easysearch.

• Security isolation is enforced through GetClusterFilter in core/elastic.go, which injects user-specific terms filters into Elasticsearch queries to restrict access to authorized clusters only.

Frequently Asked Questions

How does INFINI Console discover the version of a newly registered Elasticsearch cluster?

During registration and periodic health checks, Console invokes adapter.ClusterVersion in plugin/setup/setup.go to query the target cluster's root endpoint. This function extracts Version.Number and Version.Distribution from the cluster metadata, validating that the version meets minimum requirements (5.3+) and identifying whether the cluster runs standard Elasticsearch, OpenSearch, or Easysearch.

Can INFINI Console manage clusters running different major versions of Elasticsearch simultaneously?

Yes. The Execution Engine in modules/elastic/api/v1/ and core/elastic.go maintains per-cluster version metadata and adapts API requests accordingly. For example, the system selects appropriate document types (_doc versus doc) and index naming conventions based on the target cluster's major version, allowing a single Console instance to manage ES 1.x through 8.x clusters concurrently.

How does the system prevent users from accessing unauthorized Elasticsearch clusters?

Console implements cluster-level isolation through the GetClusterFilter function in core/elastic.go. This middleware extracts the user's permitted cluster IDs from their role assignments and injects a terms filter into every Elasticsearch query, ensuring the search results only include data from authorized clusters. This filtering occurs at the Elasticsearch query level, providing robust security enforcement.

What happens during the initial setup of INFINI Console regarding cluster management?

During first-run initialization, plugin/setup/setup.go creates a hidden system cluster identified by GlobalSystemElasticsearchID = "infini_default_system_cluster". This system cluster stores Console's own metadata in dedicated system indices (.infini_*). The setup process validates the system cluster's version using adapter.ClusterVersion and initializes version-specific templates (such as template_ilm_1_12_1.tpl for Easysearch), establishing the foundation for subsequent multi-cluster management.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}