How Easegress Health Checks Work: A Deep Dive into Backend Service Monitoring

Easegress implements health checking as a built-in component of its proxy infrastructure, using declarative specifications, concrete HTTP/WebSocket health-checker implementations, and load-balancer integration to continuously monitor and filter healthy backend services.

The easegress-io/easegress repository treats health checking as a first-class feature within its proxy and load-balancing layer. Understanding how Easegress health checks function requires examining three tightly coupled parts: the health-check specification that defines probe parameters, the health-checker implementations that execute HTTP or WebSocket probes, and the load-balancer integration that periodically evaluates backend health and rebuilds the active server pool.

Easegress Health Check Architecture Overview

The architecture separates configuration from execution. A declarative spec describes how to probe a service, while concrete checker objects handle the actual network operations. The GeneralLoadBalancer orchestrates the process by running these checks on a timer and maintaining the pool of healthy backends that the request router uses.

This design isolates probing logic from routing, enables fine-grained control over intervals, timeouts, and failure thresholds, and guarantees that traffic is only sent to services that have recently responded successfully.

Health Check Specifications in Easegress

All health-check settings are defined by HealthCheckSpec in pkg/filters/proxies/healthcheck.go. Protocol-specific specs extend this base structure to add probe-specific parameters.

Common Health Check Configuration

The base specification contains timing and threshold parameters used by all checker types:

type HealthCheckSpec struct {
    Interval string `json:"interval,omitempty" jsonschema:"format=duration"`
    Timeout  string `json:"timeout,omitempty" jsonschema:"format=duration"`
    Fails    int    `json:"fails,omitempty" jsonschema:"minimum=1"`
    Passes   int    `json:"passes,omitempty" jsonschema:"minimum=1"`
    Path     string `json:"path,omitempty"` // deprecated, HTTP uses URI
}

• Interval: How often to probe each backend.
• Timeout: Maximum time to wait for a response.
• Fails: Consecutive failures required to mark a server unhealthy.
• Passes: Consecutive successes required to mark a server healthy again.

HTTP-Specific Health Check Settings

HTTP health checks extend the base spec in pkg/filters/proxies/httpproxy/healthcheck.go with request construction and response validation parameters:

type HTTPHealthCheckSpec struct {
    Port     int               `json:"port,omitempty"`
    URI      string            `json:"uri,omitempty"`
    Method   string            `json:"method,omitempty"`
    Headers  map[string]string `json:"headers,omitempty"`
    Body     string            `json:"body,omitempty"`
    Match    *HealthCheckMatch `json:"match,omitempty"`
}

The Match field allows complex validation rules including status code ranges, header presence, and body content regex matching.

WebSocket Health Check Configuration

WebSocket health checks in the same file can combine an HTTP probe with a WebSocket handshake:

type WSProxyHealthCheckSpec struct {
    proxies.HealthCheckSpec `json:",inline"`
    HTTP                    *HTTPHealthCheckSpec `json:"http,omitempty"`
    WS                      *WSHealthCheckSpec   `json:"ws,omitempty"`
}

This flexibility allows administrators to validate both the HTTP layer and the WebSocket upgrade capability.

Health Checker Implementations

Concrete implementations of the HealthChecker interface execute the actual network probes. The interface is defined in pkg/filters/proxies/healthcheck.go:

type HealthChecker interface {
    BaseSpec() HealthCheckSpec
    Check(svr *Server) bool
    Close()
}

HTTP Health Checker

The httpHealthChecker struct in pkg/filters/proxies/httpproxy/healthcheck.go sends HTTP requests built from the specification. It uses a dedicated http.Client with configurable timeout from spec.GetTimeout().

After obtaining a response, the Match object validates status codes, headers, and body content against the configured rules. The Check method returns true only if all match conditions succeed.

Construction happens via NewHTTPHealthChecker:

func NewHTTPHealthChecker(tlsConfig *tls.Config, spec *ProxyHealthCheckSpec) proxies.HealthChecker { … }

WebSocket Health Checker

The wsHealthChecker optionally runs an HTTP health-check first, then upgrades to a WebSocket connection using gorilla/websocket.Dialer with the same timeout configuration.

After a successful dial, it validates the handshake response (HTTP 101) using the same Match validation logic. This ensures the WebSocket endpoint is both reachable and correctly performing the protocol upgrade.

Construction uses NewWebSocketHealthChecker:

func NewWebSocketHealthChecker(spec *WSProxyHealthCheckSpec) proxies.HealthChecker { … }

Load Balancer Integration and Health State Management

The GeneralLoadBalancer in pkg/filters/proxies/loadbalance.go orchestrates the health checking lifecycle. It owns a HealthChecker instance (glb.hc) and its specification (glb.hcSpec).

Periodic Health Check Execution

During initialization, the load balancer:

1. Retrieves the base spec from the checker to ensure sensible defaults for Fails and Passes thresholds.
2. Starts a ticker using spec.GetInterval() and launches a goroutine that periodically calls glb.checkServers().

ticker := time.NewTicker(spec.GetInterval())
glb.done = make(chan struct{})
glb.checkServers()
go func() {
    for {
        select {
        case <-glb.done:
            ticker.Stop()
            return
        case <-ticker.C:
            glb.checkServers()
        }
    }
}()

Server Health State Transitions

The checkServers method iterates over every backend Server:

• It calls glb.hc.Check(svr), which returns true for healthy probes and false for unhealthy ones.
• It updates svr.HealthCounter, incrementing for successes and decrementing for failures.
• When the counter crosses the thresholds defined by hcSpec.Passes or hcSpec.Fails, the server’s Unhealth flag is toggled, and the global glb.healthyServers pool is rebuilt.

Only servers where Healthy() returns true are kept in glb.healthyServers. The request routing logic (ChooseServer) always selects from this filtered list, ensuring that unhealthy backends are automatically bypassed and traffic is never routed to failed services.

Practical Configuration Example

Here is a complete YAML configuration enabling HTTP health checks for a backend service:

pipeline:
  name: http-proxy
  kind: Proxy
  spec:
    serviceName: my-backend
    loadBalance:
      policy: roundRobin
      healthCheck:
        interval: 5s
        timeout: 2s
        fails: 2
        passes: 1
        uri: /healthz
        method: GET
        match:
          statusCodes: [[200, 299]]

When this pipeline starts:

1. The LoadBalanceSpec parses the healthCheck block into a HealthCheckSpec.
2. NewHTTPHealthChecker creates an HTTP health-checker with the specification.
3. GeneralLoadBalancer.Init registers the checker, starts the 5-second ticker, and begins probing /healthz on each backend.
4. Servers that fail two consecutive probes become unhealthy and are removed from the active pool; a single successful probe marks a server healthy again.

Summary

• Easegress health checks are declarative specifications defined in pkg/filters/proxies/healthcheck.go and extended by protocol-specific structs in pkg/filters/proxies/httpproxy/healthcheck.go.
• HTTP and WebSocket checkers implement the HealthChecker interface, executing probes using http.Client or gorilla/websocket.Dialer with configurable timeouts and match rules.
• Load balancer integration occurs in pkg/filters/proxies/loadbalance.go, where GeneralLoadBalancer runs periodic checks via checkServers(), updates health counters, and maintains a filtered pool of healthy backends.
• Automatic failover ensures that only servers passing the configured Passes threshold are active, while servers reaching the Fails threshold are automatically removed from request routing.

Frequently Asked Questions

How do you configure HTTP health checks in Easegress?

HTTP health checks are configured within the loadBalance.healthCheck section of a Proxy pipeline. You must specify the interval, timeout, fails, and passes thresholds, along with HTTP-specific fields like uri, method, and optionally headers, body, and match rules for response validation. The configuration is parsed into HTTPHealthCheckSpec in pkg/filters/proxies/httpproxy/healthcheck.go.

What is the difference between the 'fails' and 'passes' thresholds in Easegress health checks?

The fails threshold defines how many consecutive unsuccessful probes must occur before a healthy server is marked as unhealthy and removed from the active pool. Conversely, the passes threshold specifies how many consecutive successful probes are required to promote an unhealthy server back to healthy status. These counters are maintained per-server in the HealthCounter field and evaluated in GeneralLoadBalancer.checkServers().

Does Easegress support WebSocket health checking?

Yes, Easegress supports WebSocket health checks through the WSProxyHealthCheckSpec defined in pkg/filters/proxies/httpproxy/healthcheck.go. This specification can include an optional HTTP health check to validate the base endpoint, followed by a WebSocket handshake validation using gorilla/websocket.Dialer. The checker validates the HTTP 101 Switching Protocols response to confirm the WebSocket upgrade capability.

How does Easegress handle unhealthy backend servers?

When a backend server reaches the configured fails threshold, the GeneralLoadBalancer sets the server's Unhealth flag to true and rebuilds the healthyServers pool, excluding the failed node. The ChooseServer method only selects from this filtered pool, ensuring traffic is automatically routed away from unhealthy backends. Once the server accumulates enough passes consecutive successes, it is reinstated into the active pool.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}