# How to Set Up Monitoring for INFINI Gateway Using Prometheus and Grafana

> Set up monitoring for INFINI Gateway with Prometheus and Grafana. This guide shows you how to scrape performance data directly from the gateway's stats endpoint for powerful visualization.

- Repository: [INFINI Labs/gateway](https://github.com/infinilabs/gateway)
- Tags: how-to-guide
- Published: 2026-03-04

---

**INFINI Gateway exposes a built-in `/stats` endpoint that renders Prometheus-compatible metrics when queried with `?format=prometheus`, enabling you to scrape performance data directly without external exporters.**

The open-source `infinilabs/gateway` repository ships with native observability features that track request throughput, latency, buffer pools, and system resources. By leveraging the built-in stats filter implemented in [`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go), you can integrate the gateway into Prometheus and Grafana within minutes.

## Enable the Prometheus Metrics Endpoint

INFINI Gateway registers its metrics module during initialization. In [`main.go`](https://github.com/infinilabs/gateway/blob/main/main.go) (lines 63-68), the framework calls `module.RegisterUserPlugin(&metrics.MetricsModule{})`, which activates the stats filter and exposes the HTTP API for metrics retrieval.

By default, the gateway listens on **port 2900** for administrative endpoints. Verify that Prometheus exposition is working by requesting the stats endpoint with the format parameter:

```bash
curl http://localhost:2900/stats?format=prometheus

```

The output returns plain-text metrics following the Prometheus exposition format:

```

buffer_fasthttp_resbody_buffer_acquired{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 1
system_cpu{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0
stats_gateway_request_bytes{type="gateway", ip="192.168.3.23", name="Orchid", id="cbvjphrq50kcnsu2a8v0"} 0

```

These counters are collected by the stats filter as it processes requests, tracking everything from buffer acquisitions to bulk-indexing statistics.

## Configure Prometheus Scraping

To collect these metrics continuously, add a scrape job to your [`prometheus.yml`](https://github.com/infinilabs/gateway/blob/main/prometheus.yml) configuration file. The critical configuration is the `params` section, which sets `format` to `prometheus` so the gateway returns the correct content type.

```yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: "infini_gateway"
    scrape_interval: 5s
    metrics_path: /stats
    params:
      format: ['prometheus']
    static_configs:
      - targets: ["localhost:2900"]
        labels:
          group: "infini"

```

Start Prometheus with this configuration:

```bash
prometheus --config.file=prometheus.yml

```

Prometheus now stores time-series data for all gateway metrics, including `system_cpu`, `system_mem`, `stats_gateway_request_bytes`, and `buffer_fasthttp_resbody_buffer_acquired`.

## Visualize Metrics in Grafana

Once Prometheus is scraping the endpoint, connect it to Grafana to build dashboards.

**Add the Data Source:**

1. Navigate to **Configuration** → **Data Sources** → **Add data source** in Grafana.
2. Select **Prometheus** and set the URL to your Prometheus server (e.g., `http://localhost:9090`).
3. Save and test the connection.

**Create Dashboard Panels:**

Use the following PromQL expressions in Grafana panels to visualize key performance indicators:

| Metric | PromQL Query |
|--------|--------------|
| **Request Throughput** | `sum(rate(stats_gateway_request_bytes[1m]))` |
| **95th Percentile Latency** | `histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le))` |
| **CPU Usage** | `system_cpu` |
| **Memory Usage** | `system_mem` |
| **Buffer Acquisitions** | `buffer_fasthttp_resbody_buffer_acquired` |

For a quick start, import this minimal dashboard JSON via **Create** → **Import**:

```json
{
  "dashboard": {
    "title": "INFINI Gateway Overview",
    "panels": [
      {
        "type": "graph",
        "title": "Request Throughput (bytes/sec)",
        "targets": [
          {
            "expr": "rate(stats_gateway_request_bytes[1m])",
            "legendFormat": "{{instance}}"
          }
        ]
      },
      {
        "type": "graph",
        "title": "Response Latency (ms)",
        "targets": [
          {
            "expr": "histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le))",
            "legendFormat": "95th percentile"
          }
        ]
      },
      {
        "type": "graph",
        "title": "CPU & Memory",
        "targets": [
          { "expr": "system_cpu", "legendFormat": "CPU" },
          { "expr": "system_mem", "legendFormat": "Memory (bytes)" }
        ]
      }
    ],
    "schemaVersion": 30,
    "version": 1
  },
  "overwrite": true
}

```

## Key Source Files and Extension Points

Understanding the source implementation helps you customize monitoring:

- **[`main.go`](https://github.com/infinilabs/gateway/blob/main/main.go)** (lines 63-68): Registers the metrics module that powers the `/stats` API.
- **[`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go)**: Implements the `filter.Filter` interface, calling `stats.Increment()` and `stats.Timing()` to record request bytes, latency histograms, and buffer usage.
- **[`docs/content.en/docs/tutorial/prometheus_integration.md`](https://github.com/infinilabs/gateway/blob/main/docs/content.en/docs/tutorial/prometheus_integration.md)**: Official documentation with additional configuration examples.

**Adding Custom Metrics:**

If you need to expose application-specific counters, modify [`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go) inside the `process()` function:

```go
stats.IncrementBy(filter.Category, "my_custom_metric", int64(value))

```

Restart the gateway after changes; the new metric automatically appears in the Prometheus endpoint output.

## Summary

- **Enable metrics**: The stats module auto-registers in [`main.go`](https://github.com/infinilabs/gateway/blob/main/main.go) and exposes `/stats` on port 2900.
- **Use Prometheus format**: Append `?format=prometheus` to receive exposition-format metrics from the stats filter.
- **Configure scraping**: Add a job to [`prometheus.yml`](https://github.com/infinilabs/gateway/blob/main/prometheus.yml) with `params: format: ['prometheus']` targeting the gateway host.
- **Visualize**: Connect Grafana to Prometheus and query metrics like `system_cpu`, `stats_gateway_request_bytes`, and `response_elapsed_ms_bucket`.
- **Extend**: Add custom counters by calling `stats.IncrementBy()` in [`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go).

## Frequently Asked Questions

### What port does INFINI Gateway use for the metrics endpoint?

By default, the gateway exposes the stats API on **port 2900**. You can verify availability by running `curl http://localhost:2900/stats?format=prometheus`. The port is configurable via [`gateway.yml`](https://github.com/infinilabs/gateway/blob/main/gateway.yml) if you need to bind to a different interface or port number.

### Do I need a separate Prometheus exporter for INFINI Gateway?

No. INFINI Gateway includes native Prometheus support through its built-in stats filter implemented in [`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go). The endpoint at `/stats?format=prometheus` returns exposition-formatted text compatible with Prometheus scrapers without requiring any sidecar exporters.

### Which metric tracks request latency in INFINI Gateway?

Request latency is tracked via the `response_elapsed_ms_bucket` histogram and related latency counters in the stats filter. To view the 95th percentile latency in Grafana, use the PromQL query: `histogram_quantile(0.95, sum(rate(response_elapsed_ms_bucket[1m])) by (le))`.

### Can I add my own custom metrics to the Prometheus endpoint?

Yes. You can emit custom counters by calling `stats.IncrementBy()` or `stats.Timing()` within the stats filter logic in [`proxy/output/stats/stats.go`](https://github.com/infinilabs/gateway/blob/main/proxy/output/stats/stats.go). Any new metrics you register will automatically appear in the Prometheus-formatted output at the next scrape interval after restarting the gateway.