What Metrics Does Easegress Expose? Complete Prometheus Monitoring Guide

Easegress exposes comprehensive Prometheus metrics covering HTTP server health, request latency, throughput, WAF blocks, AI gateway token usage, proxy connection pools, and file server cache performance, all auto-registered and available at the /metrics endpoint.

Easegress is a cloud-native traffic orchestration system that instruments its core components with detailed Prometheus metrics. The easegress-io/easegress repository defines counters, gauges, histograms, and summaries across HTTP servers, filters, and controllers in pkg/util/prometheushelper/helper.go, enabling deep observability into request processing, error rates, and resource utilization.

HTTP Server Metrics

The HTTP server implementation in pkg/object/httpserver/runtime.go (lines 480-621) defines the most extensive metric collection in Easegress. These metrics track every aspect of request processing from health status to percentile latencies.

Health and Availability

• httpserver_health – A gauge that reports 1 when the server is ready to accept traffic and 0 when it is down. Defined at line 480.

Request and Response Counters

• httpserver_total_requests – Total incoming HTTP requests (line 484).
• httpserver_total_responses – Total HTTP responses sent (line 488).
• httpserver_total_error_requests – Requests resulting in 4xx or 5xx status codes (line 492).

Latency and Size Distributions

• httpserver_requests_duration – Histogram measuring request-processing latency in milliseconds (line 496).
• httpserver_requests_size_bytes – Histogram tracking request body size in bytes (line 503).
• httpserver_responses_size_bytes – Histogram tracking response body size in bytes (line 510).

Percentile Summaries

• httpserver_requests_duration_percentage – Summary providing percentile latency (p0-p100) (line 517).
• httpserver_requests_size_bytes_percentage – Summary for request-size percentiles (line 524).
• httpserver_responses_size_bytes_percentage – Summary for response-size percentiles (line 531).

Rate and Error Gauges

Exponentially-weighted moving-average QPS metrics:

• httpserver_m1, httpserver_m5, httpserver_m15 – QPS over 1, 5, and 15 minutes (lines 538-549).
• httpserver_m1_err, httpserver_m5_err, httpserver_m15_err – Error QPS over the same windows (lines 550-561).
• httpserver_m1_err_percent, httpserver_m5_err_percent, httpserver_m15_err_percent – Error-rate percentages (lines 562-573).

Statistical Gauges

• httpserver_min, httpserver_max, httpserver_mean – Min, max, and mean latency in milliseconds (lines 574-586).
• httpserver_p25, httpserver_p50, httpserver_p75, httpserver_p95, httpserver_p98, httpserver_p99, httpserver_p999 – Detailed percentile latencies (lines 586-610).
• httpserver_req_size, httpserver_resp_size – Total request and response byte counts in the current window (lines 614-621).

Web Application Firewall (WAF) Metrics

The WAF controller in pkg/object/wafcontroller/metrics/metrics.go exposes security-specific counters.

• waf_total_refused_requests – Counter tracking the number of requests blocked by WAF rules (lines 81-86).

AI Gateway Metrics

The AI gateway controller in pkg/object/aigatewaycontroller/metricshub/metricshub.go (lines 131-163) provides LLM-specific observability.

Request Tracking

• ai_gateway_total_request – Counter for all AI gateway requests (line 131).
• ai_gateway_success_request – Counter for successfully processed AI requests (line 136).
• ai_gateway_failed_request – Counter for failed requests, labeled with the error type (lines 141-145).

Performance and Cost

• ai_gateway_requests_duration – Histogram measuring request latency for each AI provider (lines 146-152).
• ai_gateway_prompt_tokens – Counter for total prompt tokens processed (lines 154-158).
• ai_gateway_completion_tokens – Counter for total completion tokens generated (lines 159-163).

Proxy Filter Metrics

The HTTP proxy filter in pkg/filters/proxies/httpproxy/pool.go (lines 650-683) tracks connection pool performance and payload sizes.

Connection Metrics

• proxy_total_connections – Counter for all connections handled by the proxy (line 650).
• proxy_total_error_connections – Counter for connections that resulted in an error (line 653).

Payload Size Distributions

• proxy_request_body_size – Histogram for request body sizes in bytes (line 656).
• proxy_response_body_size – Histogram for response body sizes in bytes (line 663).

Percentile Summaries

• proxy_request_body_size_percentage – Summary for request size percentiles (line 670).
• proxy_response_body_size_percentage – Summary for response size percentiles (line 677).

File Server Metrics

The static file server filter in pkg/filters/fileserver/metrics.go exposes cache performance metrics for buffer pools and memory-mapped files.

Buffer Pool Metrics

• buffer_pool_hits – Counter for successful buffer-pool fetches (line 45).
• buffer_pool_miss – Counter for buffer-pool misses (line 50).
• buffer_pool_evicts – Counter for buffer-pool evictions (line 55).
• buffer_pool_ttl_evict – Counter for TTL-based evictions (line 60).
• buffer_pool_counter_size – Gauge for current number of buffers in the pool (line 65).
• buffer_pool_files – Gauge for number of files cached in the pool (line 70).

Memory-Mapped File Metrics

• mmap_hits – Counter for successful memory-mapped file reads (line 81).
• mmap_miss – Counter for memory-map misses (line 86).
• mmap_files – Gauge for number of mmap-managed files (line 91).

Accessing and Scraping Easegress Metrics

All metrics are auto-registered with the global Prometheus registry during component initialization. They are automatically exposed on the /metrics HTTP endpoint of the Easegress control plane, typically on port 9090.

Basic Scraping with curl

Retrieve the raw Prometheus exposition format:

curl http://localhost:9090/metrics

The output includes HELP text, TYPE declarations, and labeled metric values:

# HELP httpserver_total_requests Total incoming HTTP requests

# TYPE httpserver_total_requests counter

httpserver_total_requests{clusterName="default",clusterRole="primary",instanceName="instance-001",httpServerName="demo-server",kind="httpserver",routerKind="...",backend="..."} 15420

PromQL Query Examples

Calculate the 99th-percentile request latency for a specific HTTP server:

histogram_quantile(0.99, sum(rate(httpserver_requests_duration_bucket[5m])) by (le, httpServerName))

Monitor AI gateway token consumption rates:

sum(rate(ai_gateway_prompt_tokens[5m])) by (instanceName)

Track proxy error rates:

rate(proxy_total_error_connections[5m]) / rate(proxy_total_connections[5m])

Programmatic Access in Go

To read metric values in unit tests or custom tooling, use the Prometheus client library with the helper utilities:

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/megaease/easegress/v2/pkg/util/prometheushelper"
)

// Create a gauge matching the HTTP server health metric
g := prometheushelper.NewGauge(
    "httpserver_health",
    "show the status for the http server: 1 for ready, 0 for down",
    []string{"clusterName", "clusterRole", "instanceName", "httpServerName", "kind"},
)

// Set values with specific labels
g.WithLabelValues("my-cluster", "master", "node-1", "my-http", "httpServer").Set(1)

// Register with a custom registry for isolation
reg := prometheus.NewRegistry()
reg.MustRegister(g)

Prometheus Alerting Rules

Configure Alertmanager to detect HTTP server failures:

groups:
- name: easegress-httpserver
  rules:
  - alert: HttpServerDown
    expr: httpserver_health == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "HTTP server {{ $labels.httpServerName }} is down"
      description: "The HTTP server has reported a health value of 0 for more than 1 minute."

Alert on high error rates:

  - alert: HighErrorRate
    expr: httpserver_m5_err_percent > 5
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: "High error rate on {{ $labels.httpServerName }}"
      description: "Error rate is {{ $value }}% over the last 5 minutes."

Summary

• Easegress instruments HTTP servers, WAF controllers, AI gateways, proxy filters, and file servers with Prometheus metrics using the prometheushelper utility package.
• The HTTP server implementation in pkg/object/httpserver/runtime.go provides the most comprehensive coverage, including latency histograms, percentile summaries, and moving-average QPS gauges.
• All metrics are auto-registered with the global Prometheus registry during component initialization and exposed on the /metrics endpoint without additional configuration.
• Metrics cover infrastructure health (httpserver_health), security blocks (waf_total_refused_requests), LLM costs (ai_gateway_prompt_tokens), and cache performance (buffer_pool_hits).
• You can scrape these metrics with standard Prometheus configurations, query them with PromQL, or access them programmatically using the Prometheus Go client library.

Frequently Asked Questions

How do I enable metrics in Easegress?

Metrics are enabled by default in Easegress. Each component automatically registers its metrics with the global Prometheus registry during initialization. You do not need to configure anything to expose the /metrics endpoint; it is available as soon as the Easegress instance starts on its configured admin port (typically 9090).

What is the difference between histogram and summary metrics in Easegress?

Histograms (such as httpserver_requests_duration) bucket observations into configurable ranges and count events per bucket, making them ideal for aggregating latency distributions across multiple instances using histogram_quantile(). Summaries (such as httpserver_requests_duration_percentage) calculate configurable percentiles (p0-p100) using a sliding time window, providing exact quantiles for a specific instance but less suitable for aggregation across clusters.

How can I create custom metrics for my Easegress filter?

Use the prometheushelper package located at pkg/util/prometheushelper/helper.go. Import the package and use wrapper functions like prometheushelper.NewCounter(), prometheushelper.NewGauge(), prometheushelper.NewHistogram(), or prometheushelper.NewSummary(). These functions automatically handle label naming and registration with the global registry, ensuring your custom metrics appear alongside built-in ones on the /metrics endpoint.

Where are the AI gateway token metrics defined?

AI gateway token metrics are defined in pkg/object/aigatewaycontroller/metricshub/metricshub.go. Specifically, ai_gateway_prompt_tokens (lines 154-158) counts input tokens sent to LLM providers, while ai_gateway_completion_tokens (lines 159-163) counts output tokens generated. Both are counters that accumulate over time, allowing you to calculate token consumption rates using PromQL queries like sum(rate(ai_gateway_prompt_tokens[5m])).

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}