# How to Implement the Circuit Breaker Pattern for Fault Tolerance in Easegress

> Implement the circuit breaker pattern for fault tolerance with Easegress. Automatically reject requests exceeding thresholds to prevent cascading failures and improve system resilience.

- Repository: [MegaEase/easegress](https://github.com/megaease/easegress)
- Tags: how-to-guide
- Published: 2026-03-07

---

**Easegress implements the circuit breaker pattern through a Resilience Policy that wraps proxy handlers with a state machine—monitoring failure rates and response times to automatically reject requests when thresholds are exceeded, preventing cascading failures downstream.**

The circuit breaker pattern is essential for building resilient microservices architectures by preventing cascading failures when downstream services become unhealthy. In the `megaease/easegress` open-source API gateway, this fault tolerance mechanism is built directly into the proxy layer through configurable Resilience Policies. By leveraging the native circuit breaker implementation, you can protect your HTTP and gRPC traffic without writing custom middleware or modifying application code.

## How Easegress Implements the Circuit Breaker Mechanism

The core logic lives in [`pkg/util/circuitbreaker/circuitbreaker.go`](https://github.com/megaease/easegress/blob/main/pkg/util/circuitbreaker/circuitbreaker.go), which maintains a state machine with four possible states: **Closed** (normal operation), **Open** (rejecting requests), **Half-Open** (testing recovery), and **ForceOpen/Disabled** (manual override). 

The integration layer in [`pkg/resilience/circuitbreaker.go`](https://github.com/megaease/easegress/blob/main/pkg/resilience/circuitbreaker.go) bridges the declarative policy configuration with the core library by creating a `circuitBreakerWrapper`. This wrapper intercepts every request through the following lifecycle:

1. **AcquirePermission**: Validates the current state before execution. If the breaker is **Open**, it immediately returns `ErrShortCircuited`.
2. **Execute Handler**: Runs the actual request while measuring latency.
3. **RecordResult**: Feeds success, failure, or "slow call" outcomes into a sliding window (count-based or time-based).
4. **Transition State**: Evaluates thresholds (failure-rate, slow-call-rate, minimum calls) to transition between Closed → Open → Half-Open → Closed.

In [`pkg/filters/proxies/httpproxy/pool.go`](https://github.com/megaease/easegress/blob/main/pkg/filters/proxies/httpproxy/pool.go), the `ServerPool.InjectResiliencePolicy` method looks up the policy by name and injects the wrapper into the request pipeline:

```go
if name := sp.spec.CircuitBreakerPolicy; name != "" {
    p := policies[name]
    sp.circuitBreakerWrapper = policy.CreateWrapper()
}

```

When processing requests, `ServerPool.handle` wraps the handler transparently. If `ErrShortCircuited` is returned, the pool automatically generates a **503 Service Unavailable** response.

## Configuring a Circuit Breaker Policy

Define a `CircuitBreaker` resource (YAML) to specify thresholds and window behavior. The policy uses a sliding window to track recent call outcomes and determine when to open the circuit.

```yaml
apiVersion: easegress.io/v2
kind: CircuitBreaker
metadata:
  name: example-cb
spec:
  failureRateThreshold: 50          # Percentage of failures to trigger Open state

  slowCallRateThreshold: 100        # Percentage of slow calls (optional)

  slidingWindowType: "COUNT_BASED"   # Options: COUNT_BASED or TIME_BASED

  slidingWindowSize: 100            # Number of calls (count) or seconds (time) to track

  permittedNumberOfCallsInHalfOpen: 10
  minimumNumberOfCalls: 100         # Minimum calls before calculating error rates

  slowCallDurationThreshold: "1m"   # Threshold for considering a call "slow"

  waitDurationInOpen: "1m"          # Duration to wait before transitioning to Half-Open

```

**Key parameters explained:**
- **`failureRateThreshold`**: When the error rate exceeds this percentage within the sliding window, the breaker opens.
- **`slidingWindowType`**: `COUNT_BASED` tracks the last N calls; `TIME_BASED` tracks all calls within the last N seconds.
- **`minimumNumberOfCalls`**: Prevents the breaker from opening due to a single failure during low traffic.

## Attaching the Policy to Proxies

Reference the policy name in your proxy configuration using the `circuitBreakerPolicy` field. This works identically for both HTTP and gRPC proxies.

**HTTP Proxy example:**

```yaml
apiVersion: easegress.io/v2
kind: HTTPProxy
metadata:
  name: my-service-proxy
spec:
  # ... other configuration fields ...

  circuitBreakerPolicy: example-cb

```

**gRPC Proxy:**
The same injection logic applies to gRPC services via [`pkg/filters/proxies/grpcproxy/pool.go`](https://github.com/megaease/easegress/blob/main/pkg/filters/proxies/grpcproxy/pool.go). Specify `circuitBreakerPolicy` in your GRPCProxy spec to enable fault tolerance for gRPC traffic.

Deploy the configuration using the Easegress CLI:

```bash
easegressctl apply -f circuit-breaker.yaml

```

Once deployed, the server pool automatically wraps all handlers; no application code changes are required.

## Advanced Usage and Observability

For custom components or programmatic control, you can instantiate the circuit breaker directly using the internal library.

### Manual Circuit Breaker Creation

```go
import (
    libcb "github.com/megaease/easegress/v2/pkg/util/circuitbreaker"
    "time"
)

// Configure policy with 50% failure threshold and count-based window
policy := &libcb.Policy{
    FailureRateThreshold:             50,
    SlowCallRateThreshold:            100,
    SlidingWindowType:                libcb.CountBased,
    SlidingWindowSize:                100,
    PermittedNumberOfCallsInHalfOpen: 10,
    MinimumNumberOfCalls:             100,
    SlowCallDurationThreshold:        time.Second,
    WaitDurationInOpen:               time.Minute,
}

cb := libcb.New(policy)

// Wrap business logic
wrapped := func(ctx context.Context) error {
    permitted, stateID := cb.AcquirePermission()
    if !permitted {
        return libcb.ErrRejected // Request short-circuited
    }
    
    start := time.Now()
    err := myBusinessLogic(ctx)
    cb.RecordResult(stateID, err != nil, time.Since(start))
    return err
}

```

### Monitoring State Transitions

Implement an `EventListenerFunc` to capture state changes for metrics or alerting:

```go
breaker.SetStateListener(func(e *libcb.Event) {
    log.Printf("Circuit breaker transitioned from %s to %s, reason: %s",
        e.OldState, e.NewState, e.Reason)
})

```

The `Event` structure provides timestamps, previous and new state strings, and the transition reason, enabling integration with Prometheus, Grafana, or custom logging pipelines.

## Summary

- **Core Implementation**: The circuit breaker logic resides in [`pkg/util/circuitbreaker/circuitbreaker.go`](https://github.com/megaease/easegress/blob/main/pkg/util/circuitbreaker/circuitbreaker.go), while policy parsing and wrapper creation occur in [`pkg/resilience/circuitbreaker.go`](https://github.com/megaease/easegress/blob/main/pkg/resilience/circuitbreaker.go).
- **Proxy Integration**: Attach policies via the `circuitBreakerPolicy` field in HTTPProxy or GRPCProxy specs; the server pool in [`pkg/filters/proxies/httpproxy/pool.go`](https://github.com/megaease/easegress/blob/main/pkg/filters/proxies/httpproxy/pool.go) (or [`grpcproxy/pool.go`](https://github.com/megaease/easegress/blob/main/grpcproxy/pool.go)) automatically injects the wrapper.
- **State Management**: The breaker cycles through Closed, Open, and Half-Open states based on configurable failure-rate and slow-call-rate thresholds tracked in sliding windows.
- **Automatic Failures**: When the circuit is open, Easegress returns HTTP 503 Service Unavailable without forwarding requests to unhealthy backends.
- **Observability**: Hook into state transition events using `SetStateListener` for real-time monitoring of fault tolerance behavior.

## Frequently Asked Questions

### What happens when the circuit breaker opens?

When the failure rate or slow-call rate exceeds the configured thresholds, the breaker transitions to the **Open** state. In this state, `AcquirePermission` returns false, causing the wrapper to return `ErrShortCircuited` immediately. The HTTP server pool catches this error and responds with **503 Service Unavailable**, protecting the downstream service from additional load while it recovers.

### How do I choose between COUNT_BASED and TIME_BASED sliding windows?

Use **COUNT_BASED** when you want the breaker to react to error rates within a fixed number of recent requests (e.g., 100 calls), regardless of time elapsed. Use **TIME_BASED** when traffic patterns vary significantly and you want to evaluate error rates within a specific time window (e.g., the last 60 seconds), ensuring the breaker considers recent temporal behavior rather than absolute call volume.

### Can I use circuit breakers with gRPC services?

Yes. The circuit breaker implementation is protocol-agnostic at the wrapper level. For gRPC proxies, the `GRPCProxy` resource supports the same `circuitBreakerPolicy` field as `HTTPProxy`. The injection logic in [`pkg/filters/proxies/grpcproxy/pool.go`](https://github.com/megaease/easegress/blob/main/pkg/filters/proxies/grpcproxy/pool.go) mirrors the HTTP implementation, wrapping gRPC handlers to provide identical fault tolerance semantics.

### How can I force the circuit breaker to remain open or closed?

The `CircuitBreakerPolicy` supports **ForceOpen** and **Disabled** states via configuration, though typically you control this through the standard threshold parameters. For manual control in custom code, you can interact with the underlying `libcb.CircuitBreaker` instance directly and manipulate state transition logic, or simply set extreme threshold values (e.g., `failureRateThreshold: 0` to never open, or `failureRateThreshold: 1` with `minimumNumberOfCalls: 1` to open immediately on any error).