# CCX Circuit Breaker Implementation: Protecting Upstream AI APIs with Per-Key Failure Isolation

> Discover CCX circuit breaker implementation in Go. Isolate failing AI API keys with per-key failure protection, automatic circuit opening, and safe traffic restoration via exponential backoff.

- Repository: [Benedict King/ccx](https://github.com/BenedictKing/ccx)
- Tags: deep-dive
- Published: 2026-05-29

---

**CCX implements a sophisticated per-key circuit breaker pattern in Go that isolates failing AI API keys, automatically opens circuits after detecting consecutive retryable failures exceeding configurable thresholds, and uses exponential backoff with half-open probing to safely restore traffic without overwhelming unstable upstream services.**

The BenedictKing/ccx repository provides a resilient **circuit breaker implementation** in its backend Go service to prevent cascading failures when calling upstream AI APIs. Located in [`backend-go/internal/metrics/channel_metrics.go`](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go), this system tracks failure rates per individual API key (BaseURL + APIKey combination) and automatically isolates problematic endpoints before they degrade overall system performance.

## Core Architecture and State Management

### Failure Classification with FailureClass

The implementation distinguishes between retryable and non-retryable errors using the `FailureClass` enum defined at [lines 16-25](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L16-L25). Only failures marked as `FailureClassRetryable` trigger circuit breaker logic, while permanent errors like authentication failures are excluded from breaker calculations via `isBreakerRelevantFailure`.

### Three-State Circuit Design

The `CircuitState` enum ([lines 32-39](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L32-L39)) defines three distinct states:

- **Closed**: Normal operation where requests pass through to upstream services
- **Open**: Circuit is broken and requests are blocked to prevent overload
- **HalfOpen**: Probe state allowing limited traffic to test service recovery

## Threshold Configuration and Triggers

### Configurable Failure Thresholds

The circuit breaker activates based on criteria defined in constants at [lines 64-69](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L64-L69):

- `consecutiveRetryableFailuresThreshold = 3`: Opens circuit after 3 consecutive retryable failures
- Default failure rate threshold of 50% within the sliding window
- `defaultCircuitBackoffBase = 30s` with `defaultCircuitBackoffMax = 10m` for exponential backoff calculations

## State Transition Logic

### Trip Conditions (Closed → Open)

The `moveCircuitToOpenLocked` function ([lines 107-120](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L107-L120)) transitions circuits from Closed to Open when either consecutive failures exceed the threshold or the sliding window failure rate surpasses 50%. This operation sets `CircuitStateOpen`, records `CircuitBrokenAt`, calculates the next retry time `NextRetryAt` using exponential backoff, and increments `BackoffLevel`.

### Recovery Probing (Open → HalfOpen)

After the backoff period expires, `moveCircuitToHalfOpenLocked` ([lines 122-128](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L122-L128)) transitions the circuit to HalfOpen, clearing `NextRetryAt` and recording `HalfOpenAt`. The `cleanupCircuitBreakers` routine (initiated in `NewMetricsManager`) periodically traverses all keys via `advanceCircuitStateIfDueLocked` to trigger this transition automatically.

### Circuit Closure (HalfOpen → Closed)

Success in HalfOpen state triggers `handleBreakerSuccessLocked` ([lines 140-155](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L140-L155)). Once probe successes reach `halfOpenSuccessTarget` (default 1), `resetCircuitStateLocked` restores the Closed state, clears failure counters, and persists the change to storage.

### Immediate Recovery (Open → Closed)

The implementation supports rapid recovery: any successful request recorded while the circuit is Open can immediately reset the circuit to Closed via `handleBreakerSuccessLocked`, enabling quick restoration when upstream services stabilize unexpectedly.

## Persistence and Backpressure Mechanisms

### SQLite State Persistence

Circuit states survive service restarts through `persistCircuitStateLocked` ([lines 155-165](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L155-L165)), which writes to SQLite via `PersistenceStore.UpsertCircuitState`. This prevents immediate error storms following deployments or crashes. The storage schema is defined in [`backend-go/internal/metrics/sqlite_store.go`](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/sqlite_store.go) and tested in [`sqlite_store_migration_test.go`](https://github.com/BenedictKing/ccx/blob/main/sqlite_store_migration_test.go).

### Exponential Backoff Strategy

The `nextBackoffDuration` function ([lines 138-151](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L138-L151)) calculates retry delays using the formula `base * 2^level`, capped at `circuitBackoffMax` (10 minutes). Each consecutive opening of the circuit increments `BackoffLevel`, creating progressive backpressure against flapping services.

## Health Monitoring API

### Per-Key Health Verification

The `IsKeyHealthy` function ([lines 281-293](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L281-L293)) interrogates current `CircuitState` and recent breaker results, returning boolean availability for upstream routing decisions.

### Channel-Level Health Aggregation

`IsChannelHealthyWithKeys` ([lines 295-332](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L295-L332)) evaluates multiple keys simultaneously for load balancing purposes, returning true if any key in the channel remains healthy, enabling automatic failover to available endpoints.

## Practical Implementation Examples

### Recording Request Success

When an upstream request succeeds, update metrics to potentially close an open circuit:

```go
metricsMgr.RecordSuccess(baseURL, apiKey, "messages")

```

This internally calls `recordSuccessWithUsageLocked` → `appendToWindowKey` → `handleBreakerSuccessLocked`, which transitions HalfOpen circuits to Closed after `halfOpenSuccessTarget` successes.

### Recording Failures with Classification

Distinguish between retryable and permanent failures:

```go
// Counts toward circuit breaker threshold
metricsMgr.RecordFailureWithClass(baseURL, apiKey, "messages", metrics.FailureClassRetryable)

// Excluded from breaker logic (e.g., invalid auth)
metricsMgr.RecordFailureWithClass(baseURL, apiKey, "messages", metrics.FailureClassNonRetryable)

```

Only `FailureClassRetryable` increments failure counters in `handleBreakerFailureLocked` ([lines 158-168](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L158-L168)).

### Checking Key Availability Before Routing

Verify circuit state before dispatching requests:

```go
if metricsMgr.IsKeyHealthy(baseURL, apiKey, "messages") {
    // Circuit is Closed or successfully probing
    dispatchRequest(baseURL, apiKey)
} else {
    // Circuit is Open, use fallback
    useBackupChannel()
}

```

### Channel Health Broadcasting

Check if any keys remain available in a channel:

```go
activeKeys := []string{key1, key2, key3}
isAvailable := metricsMgr.IsChannelHealthyWithKeys(baseURL, activeKeys, "messages")
// Returns true if at least one key is healthy

```

### Debugging Circuit States

Inspect current breaker status for monitoring:

```go
km := metricsMgr.GetKeyMetrics(baseURL, apiKey, "messages")
fmt.Printf("State: %s, Backoff: %d, NextRetry: %v\n", 
    km.CircuitState.String(), 
    km.BackoffLevel,
    km.NextRetryAt)

```

## Summary

- **Per-key isolation**: Each BaseURL + APIKey combination maintains independent circuit state, preventing single key failures from affecting others
- **Three-state resilience**: Closed/Open/HalfOpen state machine with configurable thresholds (3 consecutive failures or 50% error rate)
- **Progressive backoff**: Exponential delay from 30 seconds up to 10 minutes prevents thundering herds during recovery
- **Persistent state**: SQLite storage via `persistCircuitStateLocked` maintains breaker status across service restarts
- **Intelligent classification**: `FailureClass` distinction ensures only retryable errors (timeouts, 5xx) trigger isolation
- **Health aggregation**: `IsChannelHealthyWithKeys` enables sophisticated load balancing across multiple keys

## Frequently Asked Questions

### What triggers the circuit breaker to open in CCX?

The circuit transitions from Closed to Open via `moveCircuitToOpenLocked` when either `consecutiveRetryableFailuresThreshold` (default 3) is exceeded, or when the sliding window failure rate surpasses 50%. These checks occur within `handleBreakerFailureLocked` after each failed request classified as `FailureClassRetryable`.

### How does CCX distinguish between temporary and permanent API failures?

The `FailureClass` enum ([lines 16-25](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L16-L25)) categorizes errors as `FailureClassRetryable` or `FailureClassNonRetryable`. Only retryable errors—such as network timeouts, connection refused, and HTTP 5xx responses—affect the circuit breaker. Authentication failures and invalid requests are excluded via `isBreakerRelevantFailure` logic.

### Can the circuit breaker survive service restarts?

Yes. The implementation writes all circuit states, backoff levels, timestamps, and failure windows to SQLite via `PersistenceStore.UpsertCircuitState`. The `persistCircuitStateLocked` function ([lines 155-165](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L155-L165)) ensures continuity across deployments and crashes, preventing immediate re-triggering of error conditions upon restart.

### What is the maximum backoff duration for half-open probes?

The exponential backoff caps at `defaultCircuitBackoffMax` (10 minutes), calculated by `nextBackoffDuration` ([lines 138-151](https://github.com/BenedictKing/ccx/blob/main/backend-go/internal/metrics/channel_metrics.go#L138-L151)) using the formula `defaultCircuitBackoffBase * 2^BackoffLevel`. Each time a circuit opens, `BackoffLevel` increments, extending the wait period until the next HalfOpen probe attempt.