# How Background-Agents Implements a Circuit Breaker Mechanism for Sandbox Failures

> Learn how Background-Agents uses a circuit breaker mechanism to prevent sandbox failures. Discover the default thresholds and how it safeguards your system.

- Repository: [Cole Murray/background-agents](https://github.com/ColeMurray/background-agents)
- Tags: deep-dive
- Published: 2026-07-13

---

**Background-Agents uses a classic circuit breaker pattern with a default threshold of 3 consecutive failures within a 5-minute window to prevent cascading sandbox spawn failures.**

The ColeMurray/background-agents repository protects its sandbox-spawning pipeline with a robust circuit breaker mechanism that throttles rapid retry attempts after repeated permanent failures. This pattern tracks consecutive spawn failures in a state object and blocks new requests when thresholds are exceeded, giving underlying infrastructure time to recover while preventing endless spawn loops. The implementation separates pure decision logic from side effects, making it both testable and resilient.

## Circuit Breaker State and Configuration

### Tracking Failures with CircuitBreakerState

Each sandbox record stores a **CircuitBreakerState** object that maintains the failure count and timestamp. In [`packages/control-plane/src/sandbox/lifecycle/decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/decisions.ts) (lines 35-42), the state interface captures two critical fields:

- `failureCount`: The number of consecutive spawn failures
- `lastFailureTime`: The timestamp of the most recent failure

### Configurable Thresholds and Time Windows

The **CircuitBreakerConfig** defines the policy boundaries. The `DEFAULT_CIRCUIT_BREAKER_CONFIG` in [`packages/control-plane/src/sandbox/lifecycle/decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/decisions.ts) (lines 58-61) establishes:

- **threshold**: 3 failures
- **windowMs**: 300,000 milliseconds (5 minutes)

These defaults balance sensitivity with stability, ensuring temporary glitches do not immediately block spawning while still protecting against persistent issues.

## Decision Logic: Evaluating the Circuit Breaker

The pure function `evaluateCircuitBreaker(state, config, now)` in [`packages/control-plane/src/sandbox/lifecycle/decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/decisions.ts) (lines 75-85) determines whether to allow or block spawn requests. It follows three discrete steps:

1. **Resets** the failure count if the time since the last failure exceeds `windowMs`
2. **Opens** the circuit when `failureCount >= threshold`, returning a `waitTimeMs` until retry
3. **Allows** the spawn to proceed if within limits

```typescript
import {
  evaluateCircuitBreaker,
  DEFAULT_CIRCUIT_BREAKER_CONFIG,
  type CircuitBreakerState,
} from "@open-inspect/control-plane/sandbox/lifecycle/decisions";

async function canSpawnSandbox(state: CircuitBreakerState): Promise<boolean> {
  const now = Date.now();
  const decision = evaluateCircuitBreaker(
    state,
    DEFAULT_CIRCUIT_BREAKER_CONFIG,
    now,
  );

  if (!decision.shouldProceed) {
    console.warn(
      `Circuit breaker open – wait ${decision.waitTimeMs!} ms before retrying`,
    );
    return false;
  }

  if (decision.shouldReset) {
    // Reset failure count in DB (not shown)
  }

  return true;
}

```

## Permanent vs. Transient Error Handling

The sandbox lifecycle manager distinguishes between error types to avoid false positives. According to the source code in [`packages/control-plane/src/sandbox/lifecycle/manager.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/manager.ts) (line 569), the circuit breaker only increments for **permanent** errors such as misconfiguration, deliberately ignoring transient network blips.

Unit tests in [`packages/control-plane/src/sandbox/lifecycle/manager.test.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/manager.test.ts) (line 930) verify that transient errors do not increment the failure counter. This selective counting ensures temporary infrastructure hiccups do not trigger unnecessary circuit breaks.

```typescript
import { SandboxProviderError } from "./provider";

try {
  await sandboxProvider.spawn(...);
} catch (err) {
  if (err instanceof SandboxProviderError && err.errorType === "permanent") {
    await db.incrementCircuitBreaker(sessionId);
  }
  // Transient errors are logged but do not affect the breaker
}

```

## Summary

- **CircuitBreakerState** tracks `failureCount` and `lastFailureTime` for each sandbox record in the database
- **Default configuration** allows 3 failures within a 5-minute window before opening the circuit
- **Pure decision function** `evaluateCircuitBreaker` handles state transitions without side effects, enabling deterministic testing
- **Permanent error filtering** prevents transient network issues from triggering false breaks, as enforced in [`manager.ts`](https://github.com/ColeMurray/background-agents/blob/main/manager.ts)
- **Implementation files** located in [`packages/control-plane/src/sandbox/lifecycle/decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/decisions.ts) and [`manager.ts`](https://github.com/ColeMurray/background-agents/blob/main/manager.ts)

## Frequently Asked Questions

### What triggers the circuit breaker to open?

The circuit opens when `failureCount` exceeds the configured `threshold` (default 3) within the `windowMs` period (default 5 minutes), as implemented in `evaluateCircuitBreaker` in [`decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/decisions.ts).

### How long does the circuit breaker stay open?

The circuit remains open until the time since the last failure exceeds the `windowMs` value (5 minutes by default), at which point the failure count resets and spawning resumes.

### Why does the mechanism ignore transient errors?

The lifecycle manager only increments the counter for permanent errors (e.g., misconfiguration) and ignores transient network blips, preventing temporary infrastructure issues from triggering unnecessary cooldown periods.

### Where is the circuit breaker logic implemented?

The core logic resides in [`packages/control-plane/src/sandbox/lifecycle/decisions.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/decisions.ts) (state definitions and decision function) and [`packages/control-plane/src/sandbox/lifecycle/manager.ts`](https://github.com/ColeMurray/background-agents/blob/main/packages/control-plane/src/sandbox/lifecycle/manager.ts) (error classification and state updates).