CCX Circuit Breaker Implementation: Protecting Upstream AI APIs with Per-Key Failure Isolation

CCX implements a sophisticated per-key circuit breaker pattern in Go that isolates failing AI API keys, automatically opens circuits after detecting consecutive retryable failures exceeding configurable thresholds, and uses exponential backoff with half-open probing to safely restore traffic without overwhelming unstable upstream services.

The BenedictKing/ccx repository provides a resilient circuit breaker implementation in its backend Go service to prevent cascading failures when calling upstream AI APIs. Located in backend-go/internal/metrics/channel_metrics.go, this system tracks failure rates per individual API key (BaseURL + APIKey combination) and automatically isolates problematic endpoints before they degrade overall system performance.

Core Architecture and State Management

Failure Classification with FailureClass

The implementation distinguishes between retryable and non-retryable errors using the FailureClass enum defined at lines 16-25. Only failures marked as FailureClassRetryable trigger circuit breaker logic, while permanent errors like authentication failures are excluded from breaker calculations via isBreakerRelevantFailure.

Three-State Circuit Design

The CircuitState enum (lines 32-39) defines three distinct states:

• Closed: Normal operation where requests pass through to upstream services
• Open: Circuit is broken and requests are blocked to prevent overload
• HalfOpen: Probe state allowing limited traffic to test service recovery

Threshold Configuration and Triggers

Configurable Failure Thresholds

The circuit breaker activates based on criteria defined in constants at lines 64-69:

• consecutiveRetryableFailuresThreshold = 3: Opens circuit after 3 consecutive retryable failures
• Default failure rate threshold of 50% within the sliding window
• defaultCircuitBackoffBase = 30s with defaultCircuitBackoffMax = 10m for exponential backoff calculations

State Transition Logic

Trip Conditions (Closed → Open)

The moveCircuitToOpenLocked function (lines 107-120) transitions circuits from Closed to Open when either consecutive failures exceed the threshold or the sliding window failure rate surpasses 50%. This operation sets CircuitStateOpen, records CircuitBrokenAt, calculates the next retry time NextRetryAt using exponential backoff, and increments BackoffLevel.

Recovery Probing (Open → HalfOpen)

After the backoff period expires, moveCircuitToHalfOpenLocked (lines 122-128) transitions the circuit to HalfOpen, clearing NextRetryAt and recording HalfOpenAt. The cleanupCircuitBreakers routine (initiated in NewMetricsManager) periodically traverses all keys via advanceCircuitStateIfDueLocked to trigger this transition automatically.

Circuit Closure (HalfOpen → Closed)

Success in HalfOpen state triggers handleBreakerSuccessLocked (lines 140-155). Once probe successes reach halfOpenSuccessTarget (default 1), resetCircuitStateLocked restores the Closed state, clears failure counters, and persists the change to storage.

Immediate Recovery (Open → Closed)

The implementation supports rapid recovery: any successful request recorded while the circuit is Open can immediately reset the circuit to Closed via handleBreakerSuccessLocked, enabling quick restoration when upstream services stabilize unexpectedly.

Persistence and Backpressure Mechanisms

SQLite State Persistence

Circuit states survive service restarts through persistCircuitStateLocked (lines 155-165), which writes to SQLite via PersistenceStore.UpsertCircuitState. This prevents immediate error storms following deployments or crashes. The storage schema is defined in backend-go/internal/metrics/sqlite_store.go and tested in sqlite_store_migration_test.go.

Exponential Backoff Strategy

The nextBackoffDuration function (lines 138-151) calculates retry delays using the formula base * 2^level, capped at circuitBackoffMax (10 minutes). Each consecutive opening of the circuit increments BackoffLevel, creating progressive backpressure against flapping services.

Health Monitoring API

Per-Key Health Verification

The IsKeyHealthy function (lines 281-293) interrogates current CircuitState and recent breaker results, returning boolean availability for upstream routing decisions.

Channel-Level Health Aggregation

IsChannelHealthyWithKeys (lines 295-332) evaluates multiple keys simultaneously for load balancing purposes, returning true if any key in the channel remains healthy, enabling automatic failover to available endpoints.

Practical Implementation Examples

Recording Request Success

When an upstream request succeeds, update metrics to potentially close an open circuit:

metricsMgr.RecordSuccess(baseURL, apiKey, "messages")

This internally calls recordSuccessWithUsageLocked → appendToWindowKey → handleBreakerSuccessLocked, which transitions HalfOpen circuits to Closed after halfOpenSuccessTarget successes.

Recording Failures with Classification

Distinguish between retryable and permanent failures:

// Counts toward circuit breaker threshold
metricsMgr.RecordFailureWithClass(baseURL, apiKey, "messages", metrics.FailureClassRetryable)

// Excluded from breaker logic (e.g., invalid auth)
metricsMgr.RecordFailureWithClass(baseURL, apiKey, "messages", metrics.FailureClassNonRetryable)

Only FailureClassRetryable increments failure counters in handleBreakerFailureLocked (lines 158-168).

Checking Key Availability Before Routing

Verify circuit state before dispatching requests:

if metricsMgr.IsKeyHealthy(baseURL, apiKey, "messages") {
    // Circuit is Closed or successfully probing
    dispatchRequest(baseURL, apiKey)
} else {
    // Circuit is Open, use fallback
    useBackupChannel()
}

Channel Health Broadcasting

Check if any keys remain available in a channel:

activeKeys := []string{key1, key2, key3}
isAvailable := metricsMgr.IsChannelHealthyWithKeys(baseURL, activeKeys, "messages")
// Returns true if at least one key is healthy

Debugging Circuit States

Inspect current breaker status for monitoring:

km := metricsMgr.GetKeyMetrics(baseURL, apiKey, "messages")
fmt.Printf("State: %s, Backoff: %d, NextRetry: %v\n", 
    km.CircuitState.String(), 
    km.BackoffLevel,
    km.NextRetryAt)

Summary

• Per-key isolation: Each BaseURL + APIKey combination maintains independent circuit state, preventing single key failures from affecting others
• Three-state resilience: Closed/Open/HalfOpen state machine with configurable thresholds (3 consecutive failures or 50% error rate)
• Progressive backoff: Exponential delay from 30 seconds up to 10 minutes prevents thundering herds during recovery
• Persistent state: SQLite storage via persistCircuitStateLocked maintains breaker status across service restarts
• Intelligent classification: FailureClass distinction ensures only retryable errors (timeouts, 5xx) trigger isolation
• Health aggregation: IsChannelHealthyWithKeys enables sophisticated load balancing across multiple keys

Frequently Asked Questions

What triggers the circuit breaker to open in CCX?

The circuit transitions from Closed to Open via moveCircuitToOpenLocked when either consecutiveRetryableFailuresThreshold (default 3) is exceeded, or when the sliding window failure rate surpasses 50%. These checks occur within handleBreakerFailureLocked after each failed request classified as FailureClassRetryable.

How does CCX distinguish between temporary and permanent API failures?

The FailureClass enum (lines 16-25) categorizes errors as FailureClassRetryable or FailureClassNonRetryable. Only retryable errors—such as network timeouts, connection refused, and HTTP 5xx responses—affect the circuit breaker. Authentication failures and invalid requests are excluded via isBreakerRelevantFailure logic.

Can the circuit breaker survive service restarts?

Yes. The implementation writes all circuit states, backoff levels, timestamps, and failure windows to SQLite via PersistenceStore.UpsertCircuitState. The persistCircuitStateLocked function (lines 155-165) ensures continuity across deployments and crashes, preventing immediate re-triggering of error conditions upon restart.

What is the maximum backoff duration for half-open probes?

The exponential backoff caps at defaultCircuitBackoffMax (10 minutes), calculated by nextBackoffDuration (lines 138-151) using the formula defaultCircuitBackoffBase * 2^BackoffLevel. Each time a circuit opens, BackoffLevel increments, extending the wait period until the next HalfOpen probe attempt.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}