How to Set Up Monitoring for INFINI Gateway Using Prometheus and Grafana

INFINI Gateway exposes a built-in /stats endpoint that renders Prometheus-compatible metrics when queried with ?format=prometheus, enabling you to scrape performance data directly without external exporters.

The open-source infinilabs/gateway repository ships with native observability features that track request throughput, latency, buffer pools, and system resources. By leveraging the built-in stats filter implemented in proxy/output/stats/stats.go, you can integrate the gateway into Prometheus and Grafana within minutes.

Enable the Prometheus Metrics Endpoint

INFINI Gateway registers its metrics module during initialization. In main.go (lines 63-68), the framework calls module.RegisterUserPlugin(&metrics.MetricsModule{}), which activates the stats filter and exposes the HTTP API for metrics retrieval.

By default, the gateway listens on port 2900 for administrative endpoints. Verify that Prometheus exposition is working by requesting the stats endpoint with the format parameter:

curl http://localhost:2900/stats?format=prometheus

The output returns plain-text metrics following the Prometheus exposition format:

buffer_fasthttp_resbody_buffer_acquired{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 1
system_cpu{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0
stats_gateway_request_bytes{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0

These counters are collected by the stats filter as it processes requests, tracking everything from buffer acquisitions to bulk-indexing statistics.

Configure Prometheus Scraping

To collect these metrics continuously, add a scrape job to your prometheus.yml configuration file. The critical configuration is the params section, which sets format to prometheus so the gateway returns the correct content type.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "infini_gateway"
    scrape_interval: 5s
    metrics_path: /stats
    params:
      format: ['prometheus']
    static_configs:
      - targets: ["localhost:2900"]
        labels:
          group: "infini"

Start Prometheus with this configuration:

prometheus --config.file=prometheus.yml

Prometheus now stores time-series data for all gateway metrics, including system_cpu, system_mem, stats_gateway_request_bytes, and buffer_fasthttp_resbody_buffer_acquired.

Visualize Metrics in Grafana

Once Prometheus is scraping the endpoint, connect it to Grafana to build dashboards.

Add the Data Source:

1. Navigate to Configuration → Data Sources → Add data source in Grafana.
2. Select Prometheus and set the URL to your Prometheus server (e.g., http://localhost:9090).
3. Save and test the connection.

Create Dashboard Panels:

Use the following PromQL expressions in Grafana panels to visualize key performance indicators:

Metric	PromQL Query
Request Throughput	sum(rate(stats_gateway_request_bytes[1m]))
95th Percentile Latency	histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le))
CPU Usage	system_cpu
Memory Usage	system_mem
Buffer Acquisitions	buffer_fasthttp_resbody_buffer_acquired

For a quick start, import this minimal dashboard JSON via Create → Import:

{
  "dashboard": {
    "title": "INFINI Gateway Overview",
    "panels": [
      {
        "type": "graph",
        "title": "Request Throughput (bytes/sec)",
        "targets": [
          {
            "expr": "rate(stats_gateway_request_bytes[1m])",
            "legendFormat": "{{instance}}"
          }
        ]
      },
      {
        "type": "graph",
        "title": "Response Latency (ms)",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le))",
            "legendFormat": "95th percentile"
          }
        ]
      },
      {
        "type": "graph",
        "title": "CPU & Memory",
        "targets": [
          { "expr": "system_cpu", "legendFormat": "CPU" },
          { "expr": "system_mem", "legendFormat": "Memory (bytes)" }
        ]
      }
    ],
    "schemaVersion": 30,
    "version": 1
  },
  "overwrite": true
}

Key Source Files and Extension Points

Understanding the source implementation helps you customize monitoring:

• main.go (lines 63-68): Registers the metrics module that powers the /stats API.
• proxy/output/stats/stats.go: Implements the filter.Filter interface, calling stats.Increment() and stats.Timing() to record request bytes, latency histograms, and buffer usage.
• docs/content.en/docs/tutorial/prometheus_integration.md: Official documentation with additional configuration examples.

Adding Custom Metrics:

If you need to expose application-specific counters, modify proxy/output/stats/stats.go inside the process() function:

stats.IncrementBy(filter.Category, "my_custom_metric", int64(value))

Restart the gateway after changes; the new metric automatically appears in the Prometheus endpoint output.

Summary

• Enable metrics: The stats module auto-registers in main.go and exposes /stats on port 2900.
• Use Prometheus format: Append ?format=prometheus to receive exposition-format metrics from the stats filter.
• Configure scraping: Add a job to prometheus.yml with params: format: ['prometheus'] targeting the gateway host.
• Visualize: Connect Grafana to Prometheus and query metrics like system_cpu, stats_gateway_request_bytes, and response_elapsed_ms_bucket.
• Extend: Add custom counters by calling stats.IncrementBy() in proxy/output/stats/stats.go.

Frequently Asked Questions

What port does INFINI Gateway use for the metrics endpoint?

By default, the gateway exposes the stats API on port 2900. You can verify availability by running curl http://localhost:2900/stats?format=prometheus. The port is configurable via gateway.yml if you need to bind to a different interface or port number.

Do I need a separate Prometheus exporter for INFINI Gateway?

No. INFINI Gateway includes native Prometheus support through its built-in stats filter implemented in proxy/output/stats/stats.go. The endpoint at /stats?format=prometheus returns exposition-formatted text compatible with Prometheus scrapers without requiring any sidecar exporters.

Which metric tracks request latency in INFINI Gateway?

Request latency is tracked via the response_elapsed_ms_bucket histogram and related latency counters in the stats filter. To view the 95th percentile latency in Grafana, use the PromQL query: histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le)).

Can I add my own custom metrics to the Prometheus endpoint?

Yes. You can emit custom counters by calling stats.IncrementBy() or stats.Timing() within the stats filter logic in proxy/output/stats/stats.go. Any new metrics you register will automatically appear in the Prometheus-formatted output at the next scrape interval after restarting the gateway.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}