# How INFINI Console Collects and Displays Cluster, Node, and Index-Level Metrics for Monitoring

> Learn how INFINI Console collects and displays cluster, node, and index-level metrics for monitoring. Discover efficient data collection and visualization techniques for Elasticsearch.

- Repository: [INFINI Labs/console](https://github.com/infinilabs/console)
- Tags: how-to-guide
- Published: 2026-03-04

---

**INFINI Console collects Elasticsearch monitoring data through agents pushing documents to a system-wide `infini-metrics-*` index, then uses time-histogram aggregations to query and transform raw metrics into visualizable time-series data via REST APIs consumed by React frontend components.**

INFINI Console is an open-source, web-based management platform for Elasticsearch clusters. Understanding how INFINI Console collects and displays cluster, node, and index-level metrics for monitoring requires examining its dual-layer architecture: a Go-based backend that aggregates time-series data from system indices and a React frontend that renders interactive dashboards using processed metric series.

## Metric Ingestion Architecture

### System Elasticsearch as the Metrics Store

The platform treats monitoring data as first-class documents. Agents (or the console itself) periodically push raw monitoring documents into the **system-wide Elasticsearch** index pattern `infini-metrics-*`. Each document contains a `metadata.name` field identifying the metric type:

- **`node_stats`**: Per-node OS, JVM, filesystem, and shard statistics (e.g., `payload.elasticsearch.node_stats.jvm.mem.heap_used_in_bytes`)
- **`index_stats`**: Per-index indexing and search counters (e.g., `payload.elasticsearch.index_stats.total.search.query_total`)
- **`cluster_stats`**: Cluster-wide document counts and resource usage
- **`cluster_health`**: Health status values (`green`, `yellow`, `red`)

The console registers a **system Elasticsearch client** (`elastic.GlobalSystemElasticsearchID`) to write and query these documents. This design decouples metric collection from the clusters being monitored, allowing the console to aggregate data from multiple Elasticsearch deployments into a single observability backend.

## Building Time-Histogram Aggregation Queries

### Metric Definition DSL

HTTP API handlers construct queries dynamically using a fluent builder pattern. In [`modules/elastic/api/v1/node_overview.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/node_overview.go), the `GetSingleNodeMetrics` function orchestrates this process:

1. **Determine time range and granularity**: `GetMetricRangeAndBucketSize` calculates the appropriate bucket size (e.g., 60 seconds) based on the query window.
2. **Define metric items**: `newMetricItem` creates containers for related metrics, while `AddLine` registers specific Elasticsearch fields and aggregation types (`max`, `sum`, `derivative`).

```go
metricItem := newMetricItem("cpu", 1, SystemGroupKey)
metricItem.AddLine("Process CPU", "Process CPU", "...", "group1",
    "payload.elasticsearch.node_stats.process.cpu.percent", "max", bucketSizeStr,
    "%", "num", "0,0.[00]", "0,0.[00]", false, false)

```

3. **Convert to Elasticsearch DSL**: `ConvertBucketItemsToAggQuery` in [`modules/elastic/api/v1/metrics_util.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/metrics_util.go) translates these definitions into nested aggregation queries. The generated DSL uses a `date_histogram` aggregation to slice data into time buckets, with sub-aggregations extracting the specified fields.

## Executing Queries and Parsing Results

### System Client Query Execution

Handlers execute queries against the metrics index using the system client:

```go
searchR1, err := elastic.GetClient(clusterID).
    SearchWithRawQueryDSL(getAllMetricsIndex(), util.MustToJSONBytes(query))

```

The raw Elasticsearch response contains nested aggregation buckets that must be flattened into time-series arrays.

### Aggregation Parsing

`ParseAggregationResult` in [`modules/elastic/api/v1/metrics_util.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/metrics_util.go) walks the aggregation tree and extracts values into a `[][]interface{}` structure where each inner slice represents `[timestamp, value]`. For health status metrics that use bucket-range aggregations, `ParseAggregationBucketResult` extracts categorical values (`green|yellow|red|offline`).

Source file reference: [`modules/elastic/api/v1/metrics_util.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/metrics_util.go) (lines 214-260) handles the generic parsing logic for bucket structures.

## Derived Metrics and Post-Processing

### On-the-Fly Calculations

Some metrics require mathematical transformation after aggregation. The backend attaches **Calc functions** to metric lines for derivations like rate calculation or latency averaging:

```go
metricItem.Lines[0].Metric.Calc = func(value, value2 float64) float64 {
    return value / value2 // Calculates average latency from total_time / count
}

```

`CollectMetricData` in [`core/insight/metric_util.go`](https://github.com/infinilabs/console/blob/main/core/insight/metric_util.go) (lines 61-84) executes these calculations after collecting raw bucket values but before serializing the response.

### Health Status Derivation

Node availability is determined by `getNodeOnlineStatusOfRecentDay`, which analyzes recent uptime values from `node_stats` documents to produce binary online/offline states. Similarly, `getNodeHealthMetric` maps Elasticsearch cluster health API responses to color-coded status indicators.

## REST API Endpoints for Metric Retrieval

### Node-Level Metrics

The `GetSingleNodeMetrics` function in [`modules/elastic/api/v1/node_overview.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/node_overview.go) returns CPU utilization, JVM heap usage, and filesystem statistics for individual nodes. The handler assembles a JSON payload containing:

```json
{
  "metrics": {
    "cpu": { "metric": { "label": "CPU", "units": "%"}, "data": [[timestamp, value], ...] },
    "indexing": { "metric": { "label": "Indexing", "units": "doc/s"}, "data": [...] }
  },
  "summary": { "node_name": "...", "status": "online" }
}

```

### Cluster and Index Metrics

`FetchClusterInfo` in [`modules/elastic/api/v1/cluster_overview.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/cluster_overview.go) provides cluster-wide aggregations including document counts, storage usage, and search latency. For per-index throughput, the `getIndexQPS` function computes indexing and search rates using **derivative aggregations** on counter fields from `index_stats` documents.

## Frontend Visualization Layer

### React Overview Components

The frontend consumes these endpoints in [`web/src/pages/Platform/Overview/Node/index.tsx`](https://github.com/infinilabs/console/blob/main/web/src/pages/Platform/Overview/Node/index.tsx) and [`web/src/pages/Platform/Overview/Cluster/index.tsx`](https://github.com/infinilabs/console/blob/main/web/src/pages/Platform/Overview/Cluster/index.tsx). The components issue requests to `/node/info`, `/cluster/info`, and `/index/qps` endpoints:

```tsx
infoAction={`${ESPrefix}/node/info?timeout=${allTimeSettingsCache.timeout || '10s'}`}

```

### Chart Rendering

The React components parse the `metrics` object from the JSON response and feed time-series data into visualization libraries (VisX or AntV). The **Overview** pages render treemaps for cluster topology, line charts for historical metrics, and status tables for node health—all driven by the aggregated data from the Go backend.

## Practical Examples

### Querying Node Metrics via cURL

Retrieve real-time node statistics and time-series data:

```bash

# Node status and summary information

curl -s "http://localhost:8080/api/v1/node/info?id=my-cluster-id&node_id=node-abc123"

# Time-series metrics (CPU, JVM, indexing rates)

curl -s "http://localhost:8080/api/v1/node/metrics?id=my-cluster-id&node_id=node-abc123&min=now-15m&max=now"

```

Both endpoints return the structured `metrics` map suitable for direct consumption by charting libraries.

### Rendering Metrics in React

The following component demonstrates consuming the node metrics endpoint:

```tsx
import React, { useEffect, useState } from 'react';
import { Line } from '@visx/shape';
import { ESPrefix } from '@/services/common';

export const NodeMetrics = ({clusterId, nodeId}: {clusterId:string, nodeId:string}) => {
  const [data, setData] = useState<any>(null);

  useEffect(() => {
    fetch(`${ESPrefix}/node/metrics?cluster_id=${clusterId}&node_id=${nodeId}`)
      .then(r => r.json())
      .then(setData);
  }, [clusterId, nodeId]);

  if (!data) return <div>Loading…</div>;

  const cpuSeries = data.metrics.cpu?.data || [];
  return (
    <svg width={600} height={200}>
      <Line
        data={cpuSeries}
        x={d => d[0]}   // timestamp (ms)
        y={d => d[1]}   // percent
        stroke="steelblue"
      />
    </svg>
  );
};

```

### Go Backend Structure

When extending the platform, handlers follow this pattern in `modules/elastic/api/v1/`:

```go
// Build query with time range and filters
bucketSize, min, max, err := h.GetMetricRangeAndBucketSize(req, clusterID, MetricTypeNodeStats, 60)

query := util.MapStr{
    "query": util.MapStr{
        "bool": util.MapStr{
            "must": []util.MapStr{
                {"term": util.MapStr{"metadata.labels.node_id": util.MapStr{"value": nodeID}}},
            },
            "filter": []util.MapStr{
                {"range": util.MapStr{"timestamp": util.MapStr{"gte": min, "lte": max}}},
            },
        },
    },
}

// Execute and parse
metrics, err := h.getSingleMetrics(context.Background(), metricItems, query, bucketSize)

```

## Summary

- **Agents push raw monitoring documents** (`node_stats`, `index_stats`, `cluster_stats`) into the system Elasticsearch index `infini-metrics-*` for centralized storage.
- **Go handlers** (`GetSingleNodeMetrics`, `FetchClusterInfo`) construct **time-histogram aggregations** using utilities in [`core/insight/metric_util.go`](https://github.com/infinilabs/console/blob/main/core/insight/metric_util.go) and [`modules/elastic/api/v1/metrics_util.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/metrics_util.go) to slice data into queryable buckets.
- **Post-processing functions** calculate derived metrics (rates, averages) and health statuses (online/offline, green/yellow/red) on aggregated results before serialization.
- **REST APIs** return structured JSON payloads containing time-series arrays (`[[timestamp, value], ...]`) formatted for direct chart consumption.
- **React frontend components** in `web/src/pages/Platform/Overview/` fetch these endpoints and render real-time dashboards using VisX charting libraries.

## Frequently Asked Questions

### How does INFINI Console store monitoring data from multiple clusters?

INFINI Console uses a dedicated **system Elasticsearch** instance (identified by `elastic.GlobalSystemElasticsearchID`) as a central metrics repository. Agents write monitoring documents with varying `metadata.name` values (`node_stats`, `index_stats`, etc.) into the `infini-metrics-*` index pattern, allowing the console to aggregate and query metrics from all managed clusters through a single client connection.

### What aggregation types does INFINI Console use for metrics?

The platform primarily uses **`max`**, **`sum`**, and **`derivative`** aggregations. **Max** and **sum** extract absolute values like CPU percentages or document counts, while **derivative** calculates rates from cumulative counters (e.g., indexing operations per second). These are configured per metric line in [`modules/elastic/api/v1/metrics_util.go`](https://github.com/infinilabs/console/blob/main/modules/elastic/api/v1/metrics_util.go) via `ConvertBucketItemsToAggQuery`.

### How does the frontend retrieve real-time metrics?

The React Overview components (located in [`web/src/pages/Platform/Overview/Node/index.tsx`](https://github.com/infinilabs/console/blob/main/web/src/pages/Platform/Overview/Node/index.tsx) and [`Cluster/index.tsx`](https://github.com/infinilabs/console/blob/main/Cluster/index.tsx)) poll the backend via endpoints like `/node/info` and `/cluster/info`. These endpoints query pre-aggregated data from the system index and return JSON containing timestamp-value pairs, which the frontend renders as time-series charts using VisX or AntV visualization libraries.

### Can I query raw metrics directly via the API?

Yes. The API endpoints expose the same data used by the UI. You can query `/api/v1/node/metrics`, `/api/v1/cluster/info`, or `/api/v1/index/qps` with time range parameters (`min=now-1h`, `max=now`) to retrieve raw time-series data. The response format contains a `metrics` object where each key holds a `data` array of `[timestamp, value]` tuples suitable for custom integrations or external monitoring systems.