How INFINI Console Collects and Displays Cluster, Node, and Index-Level Metrics for Monitoring

INFINI Console collects Elasticsearch monitoring data through agents pushing documents to a system-wide infini-metrics-* index, then uses time-histogram aggregations to query and transform raw metrics into visualizable time-series data via REST APIs consumed by React frontend components.

INFINI Console is an open-source, web-based management platform for Elasticsearch clusters. Understanding how INFINI Console collects and displays cluster, node, and index-level metrics for monitoring requires examining its dual-layer architecture: a Go-based backend that aggregates time-series data from system indices and a React frontend that renders interactive dashboards using processed metric series.

Metric Ingestion Architecture

System Elasticsearch as the Metrics Store

The platform treats monitoring data as first-class documents. Agents (or the console itself) periodically push raw monitoring documents into the system-wide Elasticsearch index pattern infini-metrics-*. Each document contains a metadata.name field identifying the metric type:

• node_stats: Per-node OS, JVM, filesystem, and shard statistics (e.g., payload.elasticsearch.node_stats.jvm.mem.heap_used_in_bytes)
• index_stats: Per-index indexing and search counters (e.g., payload.elasticsearch.index_stats.total.search.query_total)
• cluster_stats: Cluster-wide document counts and resource usage
• cluster_health: Health status values (green, yellow, red)

The console registers a system Elasticsearch client (elastic.GlobalSystemElasticsearchID) to write and query these documents. This design decouples metric collection from the clusters being monitored, allowing the console to aggregate data from multiple Elasticsearch deployments into a single observability backend.

Building Time-Histogram Aggregation Queries

Metric Definition DSL

HTTP API handlers construct queries dynamically using a fluent builder pattern. In modules/elastic/api/v1/node_overview.go, the GetSingleNodeMetrics function orchestrates this process:

1. Determine time range and granularity: GetMetricRangeAndBucketSize calculates the appropriate bucket size (e.g., 60 seconds) based on the query window.
2. Define metric items: newMetricItem creates containers for related metrics, while AddLine registers specific Elasticsearch fields and aggregation types (max, sum, derivative).

metricItem := newMetricItem("cpu", 1, SystemGroupKey)
metricItem.AddLine("Process CPU", "Process CPU", "...", "group1",
    "payload.elasticsearch.node_stats.process.cpu.percent", "max", bucketSizeStr,
    "%", "num", "0,0.[00]", "0,0.[00]", false, false)

3. Convert to Elasticsearch DSL: ConvertBucketItemsToAggQuery in modules/elastic/api/v1/metrics_util.go translates these definitions into nested aggregation queries. The generated DSL uses a date_histogram aggregation to slice data into time buckets, with sub-aggregations extracting the specified fields.

Executing Queries and Parsing Results

System Client Query Execution

Handlers execute queries against the metrics index using the system client:

searchR1, err := elastic.GetClient(clusterID).
    SearchWithRawQueryDSL(getAllMetricsIndex(), util.MustToJSONBytes(query))

The raw Elasticsearch response contains nested aggregation buckets that must be flattened into time-series arrays.

Aggregation Parsing

ParseAggregationResult in modules/elastic/api/v1/metrics_util.go walks the aggregation tree and extracts values into a [][]interface{} structure where each inner slice represents [timestamp, value]. For health status metrics that use bucket-range aggregations, ParseAggregationBucketResult extracts categorical values (green|yellow|red|offline).

Source file reference: modules/elastic/api/v1/metrics_util.go (lines 214-260) handles the generic parsing logic for bucket structures.

Derived Metrics and Post-Processing

On-the-Fly Calculations

Some metrics require mathematical transformation after aggregation. The backend attaches Calc functions to metric lines for derivations like rate calculation or latency averaging:

metricItem.Lines[0].Metric.Calc = func(value, value2 float64) float64 {
    return value / value2 // Calculates average latency from total_time / count
}

CollectMetricData in core/insight/metric_util.go (lines 61-84) executes these calculations after collecting raw bucket values but before serializing the response.

Health Status Derivation

Node availability is determined by getNodeOnlineStatusOfRecentDay, which analyzes recent uptime values from node_stats documents to produce binary online/offline states. Similarly, getNodeHealthMetric maps Elasticsearch cluster health API responses to color-coded status indicators.

REST API Endpoints for Metric Retrieval

Node-Level Metrics

The GetSingleNodeMetrics function in modules/elastic/api/v1/node_overview.go returns CPU utilization, JVM heap usage, and filesystem statistics for individual nodes. The handler assembles a JSON payload containing:

{
  "metrics": {
    "cpu": { "metric": { "label": "CPU", "units": "%"}, "data": [[timestamp, value], ...] },
    "indexing": { "metric": { "label": "Indexing", "units": "doc/s"}, "data": [...] }
  },
  "summary": { "node_name": "...", "status": "online" }
}

Cluster and Index Metrics

FetchClusterInfo in modules/elastic/api/v1/cluster_overview.go provides cluster-wide aggregations including document counts, storage usage, and search latency. For per-index throughput, the getIndexQPS function computes indexing and search rates using derivative aggregations on counter fields from index_stats documents.

Frontend Visualization Layer

React Overview Components

The frontend consumes these endpoints in web/src/pages/Platform/Overview/Node/index.tsx and web/src/pages/Platform/Overview/Cluster/index.tsx. The components issue requests to /node/info, /cluster/info, and /index/qps endpoints:

infoAction={`${ESPrefix}/node/info?timeout=${allTimeSettingsCache.timeout || '10s'}`}

Chart Rendering

The React components parse the metrics object from the JSON response and feed time-series data into visualization libraries (VisX or AntV). The Overview pages render treemaps for cluster topology, line charts for historical metrics, and status tables for node health—all driven by the aggregated data from the Go backend.

Practical Examples

Querying Node Metrics via cURL

Retrieve real-time node statistics and time-series data:

# Node status and summary information

curl -s "http://localhost:8080/api/v1/node/info?id=my-cluster-id&node_id=node-abc123"

# Time-series metrics (CPU, JVM, indexing rates)

curl -s "http://localhost:8080/api/v1/node/metrics?id=my-cluster-id&node_id=node-abc123&min=now-15m&max=now"

Both endpoints return the structured metrics map suitable for direct consumption by charting libraries.

Rendering Metrics in React

The following component demonstrates consuming the node metrics endpoint:

import React, { useEffect, useState } from 'react';
import { Line } from '@visx/shape';
import { ESPrefix } from '@/services/common';

export const NodeMetrics = ({clusterId, nodeId}: {clusterId:string, nodeId:string}) => {
  const [data, setData] = useState<any>(null);

  useEffect(() => {
    fetch(`${ESPrefix}/node/metrics?cluster_id=${clusterId}&node_id=${nodeId}`)
      .then(r => r.json())
      .then(setData);
  }, [clusterId, nodeId]);

  if (!data) return <div>Loading…</div>;

  const cpuSeries = data.metrics.cpu?.data || [];
  return (
    <svg width={600} height={200}>
      <Line
        data={cpuSeries}
        x={d => d[0]}   // timestamp (ms)
        y={d => d[1]}   // percent
        stroke="steelblue"
      />
    </svg>
  );
};

Go Backend Structure

When extending the platform, handlers follow this pattern in modules/elastic/api/v1/:

// Build query with time range and filters
bucketSize, min, max, err := h.GetMetricRangeAndBucketSize(req, clusterID, MetricTypeNodeStats, 60)

query := util.MapStr{
    "query": util.MapStr{
        "bool": util.MapStr{
            "must": []util.MapStr{
                {"term": util.MapStr{"metadata.labels.node_id": util.MapStr{"value": nodeID}}},
            },
            "filter": []util.MapStr{
                {"range": util.MapStr{"timestamp": util.MapStr{"gte": min, "lte": max}}},
            },
        },
    },
}

// Execute and parse
metrics, err := h.getSingleMetrics(context.Background(), metricItems, query, bucketSize)

Summary

• Agents push raw monitoring documents (node_stats, index_stats, cluster_stats) into the system Elasticsearch index infini-metrics-* for centralized storage.
• Go handlers (GetSingleNodeMetrics, FetchClusterInfo) construct time-histogram aggregations using utilities in core/insight/metric_util.go and modules/elastic/api/v1/metrics_util.go to slice data into queryable buckets.
• Post-processing functions calculate derived metrics (rates, averages) and health statuses (online/offline, green/yellow/red) on aggregated results before serialization.
• REST APIs return structured JSON payloads containing time-series arrays ([[timestamp, value], ...]) formatted for direct chart consumption.
• React frontend components in web/src/pages/Platform/Overview/ fetch these endpoints and render real-time dashboards using VisX charting libraries.

Frequently Asked Questions

How does INFINI Console store monitoring data from multiple clusters?

INFINI Console uses a dedicated system Elasticsearch instance (identified by elastic.GlobalSystemElasticsearchID) as a central metrics repository. Agents write monitoring documents with varying metadata.name values (node_stats, index_stats, etc.) into the infini-metrics-* index pattern, allowing the console to aggregate and query metrics from all managed clusters through a single client connection.

What aggregation types does INFINI Console use for metrics?

The platform primarily uses max, sum, and derivative aggregations. Max and sum extract absolute values like CPU percentages or document counts, while derivative calculates rates from cumulative counters (e.g., indexing operations per second). These are configured per metric line in modules/elastic/api/v1/metrics_util.go via ConvertBucketItemsToAggQuery.

How does the frontend retrieve real-time metrics?

The React Overview components (located in web/src/pages/Platform/Overview/Node/index.tsx and Cluster/index.tsx) poll the backend via endpoints like /node/info and /cluster/info. These endpoints query pre-aggregated data from the system index and return JSON containing timestamp-value pairs, which the frontend renders as time-series charts using VisX or AntV visualization libraries.

Can I query raw metrics directly via the API?

Yes. The API endpoints expose the same data used by the UI. You can query /api/v1/node/metrics, /api/v1/cluster/info, or /api/v1/index/qps with time range parameters (min=now-1h, max=now) to retrieve raw time-series data. The response format contains a metrics object where each key holds a data array of [timestamp, value] tuples suitable for custom integrations or external monitoring systems.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}